Imaging Systems Evan Darling
1
Dear Reader, The contents of this book were created for the Imaging Systems class at the Rochester Institute of Technology in the Fall 2013 Semester. While I may not be with you to explain the many intricacies of imaging, editing, and displaying content, I have created this book to give as much detail on the subject as is possible. In addition to the information on imaging systems, additional information will be given about microscopy. As photomicrographs (photos taken through a microscope) are shown in Imaging Systems chapters, they will be referenced in the microscopy section. Microscopy is the field of using a microscope to record information about a sample. At the time of this writing, I am a Junior at RIT majoring in Photographic Sciences, and Biology with a minor in Imaging Systems and a concentration in German. This book is dedicated to all those to whom I owe a facet of my understanding and intrigue. I would not be without a twinkle in my eye without those who have helped it get there. Thank you for taking the time to read my book. If you have any comments, questions, or concerns, feel free to contact me. Sincerely, Evan Darling
Rochester Institute of Technology Photographic Sciences, Biology ejd1470@rit.edu 413-475-2460
2
3
4
Table of Contents 4
Resolution
14
The Imaging Pipeline
18
Input
22
Process
26
Output
32
Imaging Systems In Microscopy
5
6
Resolution
7
Resolution
Stereocilia, resolved through the use of computational super-resolution microscopy.
There are four different kinds of resolution. • • • •
Spatial Tonal Color Temporal
Resolution is the ability to be able to discern two structures as seperate. These four resolutions are all incorporated into a moving picture, while temporal resolution is excluded from a photograph. Because of this and the nature of this book, temporal resolution will be excluded, other than the brief explaination at right. 8
Temporal Resolution The ability to record two moments of time as separate is both the holy grail and chalice of death to the motion picture industry. The frame rate at which a movie camera records and stores each still image it takes determines a variety of effects when viewed. Temporal and spacial resolution trade off due to computational processing and storage. A higher spatial resolution = a lower temporal resolution. As the motion picture science industry continues to grow, camera systems with greater spatial to temporal resolution relationships are made.
Calculating Image Size
A 4-channel confocal microscopy image of a rat cochlea
Spatial x Tonal x # of channels = file size, in bytes To calculate image size, the various kinds of resolution must be multiplied. The image shown above is a 2 mega-pixel image, with a spacial resolution of 1024 pixels x 1024 pixels. It has 3 color channels and 8 bits of tonal resolution. Given these facts, file size can be calculated as seen at right.
Type Of Res. Spatial X Spatial Y Tonal Color Channels
# 1024 1024 8 3 25,165, 824 Bytes /1024 76 KiloBytes /1024 24 MegaBytes 9
Tonal Resolution Original
32-bit
16-bit
4-bit
2-bit
A photomicrograph of a contact lens, displaying the variations in tonal resolution levels
The tonal resolution of a recording or display medium can be represented by its bit depth. Bit depth refers to the number of tones that are capable of being stored or displayed. Bits are calculated on a factor of 2 basis. For example, an 8-bit (2^8) image will reproduce 256 tones. RAW files that come out of a digital camera are typically 14-bit images. This allows for a great amount of leway when editing the image in post-processing. As seen above, images that do not have a high tonal resolution will show banding. 10
Various file formats all have different amounts of tonal resolution that they truncate their files to, if at all. For example, JPEG files are all between 5-bit and 8-bit. To the human eye, this is an acceptable amount of tones to create smooth tonal gradations. If one were to save a JPEG file in photoshop, one would be able to determine the ‘quality’ of the file being saved. This slider moves between the perceptual gray area between 5-bit and 8-bit tonal resolution, among other changes that can be made.
Tonal resolution, displayed by bit depth
Tonal resolution, displayed as histograms
The tonal bands above display how smooth the tones in a photograph will be reproduced at that level of tonal resolution. As can be seen in the 32-level image and gradient, the smooth changes between tones are barely recognizable. Using this knowledge, coupled with the drive to save storage space, most print magazines print their images as 5-bit. However, there are occasions when this becomes noticeable in the print.
The Histogram
The histograms in the right column display the contrast differentiation achieved by each variation in tonal resolution. Histograms are typically displayed as having tones between 0 and 255.
Histograms are a common way for photographers to quickly analyze their digital photographs to make sure that there are no under/over-exposed parts of the image. While many accept their histograms give the final ruling on exposure, they can be misleading. Histograms are represented by values between 0-255, which gives 256 possible tones (8-bit). Depending upon the actual tonal range of the image and the histogram-display algorithm of the software the image is being displayed in, the histogram may look different. 11
Spatial Resolution The images at left show various rendering intents for resizing images. This shows how spatial resolution can be utilized and manipulated. The original, at right, was scaled up 150% for each mode of interpolation.
Crystallized menthol under polarized light
Nearest neighbor interpolation averages the pixel being affected with the nearest point. This method of resampling preserves hard edges, but also causes a significant amount of pixelation.
Bilinear interpolation uses the closest 2x2 block of pixels around the pixel being edited. It then takes a weighted average of the 4 pixels to get its final value. This interpolation method is similar to bicubic resampling, as it is used to achieve a smooth change between pixels when scaling up and image.
Bicubic interpolation averages the pixel being affected with a surrounding 4x4 grid (16 pixels total). This results in smooth transitions between pixels and less interpolation artifacts.
12
Color Resolution Color resolution can best be described as the number of different areas in which tonal values reside. The polarized menthol crystals at right are displayed in the RGB color mode. Tonal resolution determines how many colors can be reproduced within a certain color palette.
Crystallized menthol under polarized light
The photograph at left displays the original image in the absence of the Blue color channel.
The photograph at left displays the original image in the absence of the Green color channel.
The photograph at left displays the original image in the absence of the Red color channel.
13
The Imaging Pipeline
14
15
The Imaging Pipeline A visual representation of how a photograph is processed In Camera 1
2
A lens collects light and allows it to strike the imaging sensor of the camera
An analog-to-digital converter turns light signals into voltage counts in pixels
In RAW Processor 5
6
Data stored in a file format container
CFA interpolation/ Demosaicing
Post-Processing 9
16
10
Lens and sensor corrections and adjustments
Exposure and tonal adjustments
3
4
Defected pixels are corrected for
7
Photo response non-uniformity
8
Neural balance is applied
11
Gamma correction is applied
12
Artistic manipulations
Output
17
Input
18
19
Input CCD Sensors Charge-Couple Device sensors are used in a wide variety of applications, from consumer cameras to high-end medical and quantitative imaging devices. As photons strike the sensor, their energies are collected by a capacitor, which stores a voltage in a photo-activated well, more commonly known as a pixel. The array of these semiconductors is read out by a shift register. There are two ways that the register interracts with the pixel wells.
Interline Transfer
CCDs that have a shift register that reads out one line of pixels at a time are known as interline transfer. First, the sensor passively collects photons. Once the exposure is over, each line of the sensor is sequentially read out. As this occurs, the pixel lines still accumulate charge, leading to artifacting.
Frame Transfer
A CCD that builds a charge and then reads out every photosite simultaneously is known as frame transfer. A portion of the sensor remains hidden from light, while the other collects light. The regions are switched, and the portion of the sensor that is hidden can be read out. This removes a majority of the artifacting that occurs with interline transfer, but also drives the cost up.
EMCCDs
Electron-multiplying CCDs have a gain register in between the shift register and the output amplifier. The operator of the camera is able to adjust the number of stages that the signal passes through the gain register. This allows for a great amount of signal improvement, with negligent amounts of additional noise. EMCCDs are very commonly used in microscopy due to their high quantum efficiency (QE). A camera’s QE can be expressed as a percentage of the number of photons that are actively converted into a voltage. Photographic film has a QE around 10%, while typical CCDs have a QE around 90%. 20
CMOS Sensors
Active-pixel Sensors (APS) have a photodiode with an amplifier as their basic composition. CMOS sensors are the most common type of APS. They are used in many applications, from consumer cameras to high-speed military-grade cameras. Compared to CCD sensors, they are cheaper, consume less power, and can have the sensor and image processing functions on the same circuit. They are much less consistent, and have a shorter life-span than most CCDs, however. In consumer cameras, CMOS sensors are typicaly arranged in a Bayer pattern, although there are other Color Filter Arrays (CFAs) available, that meet specific needs. Other CFAs include the RGBW sensor pattern, CYGM sensor pattern, and Foveon.
Bayer Pattern
CMYG Pattern
RGBW Pattern
Foveon x3
The foveon x3 sensor type is a unique photo-sensor, in that it has 3 layers of pixels for each additive color channel. Each R, G, and B layer is sensitive to its respective color by the ability of the light to focus on that particular layer. This is possible due to the fact that different wavelengths of light will focus at different points due to the inherent properties of their wavelength. The foveon has the additional advantage of not needing to be demosaiced. Also, because the sensor does not have pixels that transmit only 1 channel side-by-side, it has tree times the sensitivity in the Blue and Red channels and twice the amount of sensitivity in the green channel as a sensor that has a Bayer pattern.
Scanners
The diagram at right shows how objects placed on a scan bed can be imaged. As the lamp moves across the flatbed, the light it emits reflects off of the object. This light is reflected by a mirror and redirected onto a ccd sensor. The information that the ccd captures is then converted to a digital signal and able to be read. More information on determining scan resolution can be found on in the output section on page 23. 21
Process
22
23
Process Image Processing The on-board image processor that all cameras have is a crucial key in being able to form the image that has been taken. Image processors can perform a range of tasks that are both inherent to the camera, and necessitated by the settings of the user. The processor takes away some of the inherent defects of CMOS sensors, such as noise caused by the sensor’s architecture. The processor will always perform demosaicing, sensor array transformation, noise reduction, and some amount of image sharpening. How these inherent functions are carried out, and the extent of them, are determined by the manufacturer, as is stored in the firmware of the camera. As image processors continue to advance, a variety of parameters will be improved: • • • • • • •
Canon’s ever-changing image processors continue to out-perform the previous iterations. The Digic 5+ processor, implemented in the 5d Mark III, boasts a host of improvements over the Digic 4. The Digic 5+ is 17x faster, has 75% more noise reduction, and supports 60fps 1080 Video.
Nikon’s Expeed image processors also boast comparable improvements between generations to the Canon processors.
greater sensitivity to light lower noise faster acquisition rates faster processing speeds tonal reproduction image stabilization lens and sensor corrections
Demosaicing Carried out by an algorithm that is implemented in the image processor, demosaicing creates a full-color image from the data that the Color Filter Array collects off of the imaging sensor. The values of each pixel are determined via Mean Interpolation, a method of affecting a single pixel by its surrounding pixels.
Bayer Pattern Filter
Pixel Demosaicing
24
Color Space There are a variety of color spaces that, made by differing industries trying to achieve a standard, allow a user to edit a photograph very specifically. Each is best used for a specific purpose. This purpose can be determined by a variety of factors, ranging from the desired affect while editing to the requirements of the output medium. Being knowledgeable on a few is helpful in the process of handling an image.
RGB
The RGB color space is the most commonly used. The user is capable of interracting with the image as it was captured and stored in the camera. It is the additive color model, meaning that if 100% Red, Green, and Blue are added together, white will be produced. The absence of color is black.
CIE L*a*b*
The CIE L*a*b* color space allows the photographer to edit 3 channels of luminance, red-green, and blue-yellow. This color space was designed to match the perception of human vision.
CMYK
The CMYK color space was designed for physical output devices, such as printers. Being a subtractive color space, when all of the color channels are added, black is produced. This is especially helpful in saving ink, as the white of the paper is shown when little or no ink is applied.
LMS
The Long, Medium, Short color space matches the 3 types of color-sensing cones in the human eye. each parameter of the color space is meant to correlate to the way that human vision would perceive the scene in response to long, medium, and short wavelengths of light. As such, it is based off of the RGB color space. It is used when performing chromatic adaptation, the act of making an object look the same under different lighting conditions.
25
Output
26
27
Output Ink-Jet Printers
There are two categories of Ink-Jet printers: Continuous and Drop-On-Demand. Ink-Jet printers are very common for consumer use and general desktop printing. Continuous printers always drop dye, but when it not needed to be applied to the paper, it is directed into a catcher and recycled in a resevoir. Drop-On-Demand printers only apply ink drops when required to. The application process is either thermal, mechanical, magnetic, or electrostatic.
Laser Printers
A xerographic printing operation is employed in laser printers. A laser is scanned across the printer’s photoreceptor and an image is produced. In order for the printer to know how to create the image, data must be processed through a Raster Image Processor (RIP) that is proprietary to the printer. When the ink is applied to the paper, it is applied to make a halftone image. The information sent to the printer is encoded by a page description language.
Dye Sublimation
High quality, continuous tone images are produced by dye sublimation printers. Special papers that contain dyes that are used for printing must be used. After the printer processes the data sent to it, special lasers are used to induce the dyes in the paper to sublimate. Sublimation is the process of inducing materials suspended in a solid state to become gaseous and skip their liquid state. In order to protect these prints, special coatings are alse used.
Offset Printing
The technique of printing an image to a photographic plate that is manipulated by a rubber blanket and applied to a substrate is known as offset printing. This fast method of printing is used for large-scale production volumes, like newspapers and most books.
28
Cathode Ray Tubes
A vaccuum-sealed tube with an electron gun that projects onto a fluorescent screen is known as a cathode ray tube. There are a few steps that are taken to forming an image on the fluorescent screen. First, the electron gun is turned on and electrons are accelerated toward the fluorescent screen. Without anything to force the electrons into a specified pattern, the image that projects onto the screen in ‘static.’ Second, data from the input device is sent to magnetic coils that manipulate the stream of electrons into forming an image. Once the electrons arrive at the fluorescent screen, they excite phosphors that burn and give off specific colors for a small amount of time. The electron gun sweeps across the fluorescent screen 30 times a second, which is an adequate temporal resolution for the human eye to not notice it happening. After all of this, an image is formed and the observer(s) are, hopefully, entertained.
Liquid Crystal Displays
Several modes of outputting light from a screen have replaced the conventional implementation of a CRT. Liquid crystal displays (LCDs) are more energy efficient, cost-effective, thinner, and last longer. White light that is unpolarized, meaning that it does not travel at any specific angle, becomes polarized so that it does travel at a specific angle. The light then passes through a series of filters to reach a matrix of liquid crystals. These crystals orient themselves in specific directions, dependent upon a voltage that is applied to them. This voltage (the ITO film layers apply this) is dependent upon a computer that processes input signals. By passing through the liquid crystals, the angle of the light that came through the first polarizer is changed. The light is then passed through R, G, or B filters and polarized again. Once colorized, the light is put onto a glass substrate that an observer can view. The diagram at right shows a cross-section of a single LCD pixel and how it interacts with light. 29
Output DPI Vs PPI
Dots Per Inch and Pixels Per Inch are both measures of spacial resolution of output mediums. PPI, sometimes referred to as Lines Per Inch (LPI), is used when talking about the resolution capabilities of continuous tone devices. These devices can produce gray levels as a measure of lightness. Examples include computer monitors, scanners, digital cameras, and dye-sublimation printers. Dots Per Inch refers to the number of ink drops a printer can place within a square inch. Because printers cannot vary lightness by adjusting the intensity of light being emitted, they need to vary lightness by applying more/less dots within a certain area. When determining the quality of a printer or how well a printer will be able to reproduce a photograph, a simple conversion can be utilized.
Calculating scan resolution
In the input section, how a scanner works is shown (page 17). In order to determine the resolution at which the scanner needs to do its duty, one must use the formula at right. 2 is added to the formula to allow for some wiggle room in the printing of the image.
First, one must consider what she/he is printing to. If printing to a dye-sublimation printer that is capable of continuous tone, no conversion is necessary. If it is necessary to convert dots to pixels, the rule of 16 can be used. To do this, simply divide the DPI that the printer being used is capable of, by 16, to find the PPI it is can support. 16 is used because it is most common for an image to be 8 bits, meaning that it has 256 gray levels. Because a pixel is square, we can find the square root of the number of gray levels needed to be reproduced. The square root of 256 is 16. When a different bit level is being used, the following formula can be used: PPI =
DPI √ Dots/pixel
2 x Magnification desired x Output Resolution Desired / (√ Dots/pixel) = Scanning Resolution Needed
Example An old 35mm negative needs to be scanned. You will be printing it on an 8.5x11 inch piece of paper at 8 bits and 300dpi. What resolution will it need to be scanned at in order to be printed well? 35mm film is actually 24x36mm, which is .945x1.417 inches. If the long dimension of the print is divided by the long dimension of the film, the answer is 7.76. Because we will be printing the scanned image at 8 bits, we can apply the rule of 16. When calculated, the final scanning resolution needed is 291ppi. This is, however, not commercially available, so it is alright to round up to 300ppi.
30
2 magnification = 7.76 Resolution Desired = 300dpi / 16 = 291ppi round up 300ppi
A micrograph of Daphnia Magna, processed to show halftoning
Printers have significantly less colors available than continuous tone devices, such as computer monitors. In order to produce the perception of the same amount of colors as are seen on a screen, most printers employ a halftone process. To halftone a continuous-tone image, which theoretically has an infinite range of colors, an image is reduced to a binary image consisting of a series of dots. These dots are either Cyan, Magenta, Yellow, or Black (CMYK color). The lightness of a certain area of a halftoned image is determined by the size of the ink dot that is applied to the paper.
Scanning Line Art
When scanning something that does not have a range of tones that are needed to be reproduced, it is necessary to have a different scanning resolution. Line art, the opposite of something that is continuous tone, is binary, meaning that it is either black or white. To scan text and line art properly, one must remove the need to divide DPI by the square root of the dots/ pixels. This will result in a much higher scanning resolution necessary.
2 x Magnification Desired x Output Resolution Desired = Scanning Resolution
31
Microscopy
32
33
Microscopy
“I saw nothing,” said Watson. “That is because, my friend, you were too busy looking at the scenery,” said Holmes, who was putting his walking stick to good use, stabbing the ground at almost every step. “You take pleasure in the big picture, the grand view, the distant prospect. You are a tourist, Watson, whereas I am more interested in the small and particular, for the world is never more revealing than when it is studied in intimate detail, “ said Holmes. -Larry Millett Sherlock Holmes and The Red Demon (1996)
34
A stitched micrograph of the intestinal tract of an earthworm. Combining images by photographic ‘stitching’ is a common way to increase spatial resolution without sacrificing other kinds.
35
The Scale Of Things A visual representation of scale, as shown on this page at 100x magnification
.1
er et
m
es
illi
s
s er et
m
er et m ro
no
na
ic
ch
in
m
m
00
s er rs et te m me 54 illi 02 m .0 .54 2
1
0
,0
10
10
36
Optical Resolution When imaging anything, the image collected is a likeness of the scene observed, just diffracted by the lens used to focus the light into the imaging sensor. When photographing a spot of a certain size, the spot will appear to have a hazy ring around it. This ring is a product of the diffraction of the lens. The size of the spot is related to the resolution. Resolution is being able to tell the difference between two closely positioned bright objects, and one big object. If two objects are closer together than your resolution, then they blur together in the microscope image and it’s impossible to tell that they are two points. The best resolution for an optical microscope is about 0.2 microns = 200 nm.
In certain applications, it is necessary to use high NA objectives and other specific pieces of equipment. The diagram below describes the NA of an objective in relation to its ability to collect light, where n is the refractive index of the medium in between the objective and the specimen and u is the angle of acceptance.
Original spot diffracted spot two spots, resolved seperately
two spots, barely resolved separately two spots, not resolved separately
To be able to resolve structures that are of a certain size, there are many components of a microscope system that must work in harmony. It is sometimes necessary to use objectives of high numerical aperture (NA).Most of the time, however, because the specimen is readily resolved with use of lower numerical aperture objectives, it may not be. Understading this is particularly important because high numerical aperture and high magnification are accompanied by the disadvantages of very shallow depth of field (this refers to good focus in the area just below or just above the area being examined) and short working distance. Thus, in specimens where resolution is less critical and magnifications can be lower, it is better to use lower magnification objectives of modest NA in order to yield images with a higher working distance and depth of field.
The resolution of a microscope objective is defined as the smallest distance between two points on a specimen that can still be distinguished as two separate entities. Numerical aperture determines the resolving power of an objective, but the total resolution of a microscope system is also dependent upon a number of other components of the microcscope system. The higher the numerical aperture of the total system, the better the resolution.
Correct alignment of a microscope’s optical system is of paramount importance to ensure maximum resolution. The wavelength spectrum of light used to image a specimen is also a determining factor in resolution. Shorter wavelengths are capable of resolving details to a greater degree than are longer wavelengths. To determine the resolution (R) of the system per the wavelenth (λ) of light being used, the equation below can be used. R = λ/(2NA) 37
Special Microscopy Super-resolution imaging
Stereocilia, resolved through the use of computational super-resolution microscopy
Super-resolution
Stereocilia
The resolution of any image is, ultimately, limited by the diffraction caused by the lens. Through the use of image analysis algorithms that remove some of the diffraction imperfections of the lens, greater optical resolution can be achieved. This is known as de-convolution. Conversely, convolution algorithms are used in typical image manipulation. For example, the sharpness of an image can be adjusted by using a convolution algorithm.
Stereocilia are small hair-like projections used as the final mechanical component in a series of energy transfers that allow animals to hear. Below the timpanic membrane (eardrum) of is a fluid-filled space. The stereocilia project into this fluid. As vibrations in the air cause the timpanic membrane to vibrate, the fluid moves, along with the stereocilia. When they move, they send a signal to the brain. These signals are processed and percieved as ‘sound.’ Arranged in small v-shaped bundles, stereocilia are approxamately 120 nanometers wide, which is below the diffraction limit of optical light microscopy.
The stereocilia in the photograph above are capable of being viewed at such large magnification on the page because they were imaged with a 5.5 mega-pixel camera. 38
Confocal Microscopy
A 4-channel confocal microscopy image of a rat cochlea
Confocal Microscopy The name “confocal� stems from the ability of the system to detect only light produced by fluorescence very close to the focal plane. A confocal microscope uses point illumination and a pinhole in optically corresponding planes in front of the detector to eliminate out-of-focus light. The point source is a laser that moves across the field of view and excites the fluorescent specimen at the focal plane. This is especially useful when imaging specimens that are particularly thick or are very dense. The diagram at right shows how a confocal microscope illuminates and detects the fluorescence of a specimen. 39