Imaging Systems Karie Eng
Contents 6 8 10 12 13
Fundamentals Spatial Resolution Tonal Resolution Spectral Resolution Temporal Resolution File Formats
16 17 18 20 21
Input Sensors Color Filter Array RGBW Demosaicing CCD vs CMOS
24
Process Image Processing Pipeline
28 29 30
Output PPI and DPI Halftoning and LPI Printer Technologies
1 Fundamentals
Spatial Resolution Resizing of an image can be done in different ways. Here we observe the similarities and differences between three types of interpolation. Interpolation means the individual pixels in the image are resampled to either increase or decrease the dimensions of an image. The original file has been resized to 150% using the three types of interpolation.
Original File
6
Chapter 1
Nearest Neighbor Interpolation
The image to the left has been resized using Nearest Neighbor Interpolation. This algorithm involves duplicating the pixels next to each other. The biggest advantage is the speed. It took 0.4s to complete this type of interpolation for the original image. This type of interpolation is best for preserving a file with hard/straight edges. Pixelation occurs when using this process on photographic content as can be seen in the magnified image. The image to the left has been resized using Bilinear Interpolation. This algorithm involves the averaging of pixels in each area. It took 1.4s to complete this type of interpolation for the original image.
Bilinear Interpolation
The image to the left has been reized using Bicubic Interpolation. This algorithm involves the averaging of pixels in each area like the latter interpolation, but uses a more complex equation. It took 1.6s to complete this type of interpolation for the original image. This type of interpolation boasts the smoothest gradations. Bicubic Interpolation
Fundamentals
7
TonalResolution Tonal Resolution of an image refers to the number of gray levels it contains. The gray levels are the amount of individual tones between the lightest and the darkest tones. In a grayscale image this would be between white (0) and black (255). Color photographs contain the 3 channels red, green, and blue. To calculate the number of levels in each channel you need to calculate bit depth. The equation for bit depth is 2 # bits/pixel = # levels per channel. For instance an 8-bit image is 28 = 256 levels.
8 bits/pixel, 256 gray levels
5 bits/pixel, 32 gray levels
4 bits/pixel, 16 gray levels
3 bits/pixel, 8 gray levels
2 bits/pixel, 4 gray levels
8
5 bits/pixel, 32 gray levels
Histograms help determine the tonal quality of an image. 0 represents black, the darkest part of the image, and 255 represents white, the brightest part of an image. The distribution of pixels on the histogram shows the overall tonal quality. For instance, the different shapes of the histograms below illustrate how an image will appear.
4 bits/pixel, 16 gray levels
3 bits/pixel, 8 gray levels
2 bits/pixel, 4 gray levels
Fundamentals
9
Chapter 1
8 bits/pixel, 256 gray levels
As seen in this example, as the number of bits/pixel increases the image becomes more aesthetically pleasing. The smoothest transition between tones can be seen in 8 bits/ pixels or 256 levels which is the highest range in shades the human eye can distinguish. This is the best looking image because tones appear the most continuous.
SpectralResolution Spectral resolution, or in other words color resolution, refers to the channels that make up an image. Typically cameras and monitors use the Additive Color System consisting of RGB color space and printers use the Subtractive Color System consisting of CMYK color space.
Additive Color System Cameras capture images through three monochrome signals. These make up the three channels of filters, red, green, and blue. The brightness in each channel is affected by the amount of light that passes through each color filter. Each filter blocks all colors of light except its own. For instance, a red filter will only allow red wavelengths of light to pass through. Colors are created by combining these additive colors. White is created when mixing red, green, and blue. Cameras and monitors use RGB color space.
10
Red + Green = Yellow Green + Blue = Cyan
Printing is achieved by the mixing of dyes, inks, and pigments. Cyan, magenta, yellow, and black make up the color space used in printing. Black is created when mixing cyan, magenta, and yellow.
Cyan + Magenta = Blue Magenta + Yellow = Red
Fundamentals
11
Chapter 1
Subtractive Color System
Temporal Resolution Temporal resolution defines the refresh rate on a monitor when displaying a movie. The faster the refresh rate, the better the image will look. The acceptable refresh rate in modern TV and movies is between 24 – 48 frames per second. High-speed cameras record between 50 – 300 frames per second or more.
Still taken from an HD movie shot at 24 frames per second
12
Mimimum JPEG compression, 114 kilobytes
Image file formats are the standard means of storing data. There are various file formats designed for specific purposes. Lossless compression keeps all of the data from the original file meaning no information is lost. Lossy compression eliminates redundant or unnecessary pieces of information for in order to reduce file size. TIFF is a popular file format and JPEG is a popular type of compression.
TIFF characteristics -8 bits/channel (24 bits of color) -Lossless image compression -RGB color space
Medium JPEG compression, 214 kilobytes
JPEG characteristics -8 bits/channel (24 bits of color) -Lossy compression -Better suited for continuous tone images
Maximum JPEG compression, 613 kilobytes
Fundamentals
13
Chapter 1
File Formats
2 Input
Sensors Digital cameras and analog cameras both are sensitive to light in order to record information and produce a final image. However, digital cameras use a sensor and analog cameras use film. These two methods are related.
Film Film is coated with a gelatin emulsion containing light-sensitive silver halide crystals. When exposed to light the film goes through a slight chemical reaction that produces an invisible latent image. The film is then developed in order to make this latent image fully visible. Color film typically uses layers on top, red, green, yellow, and blue.
Digital Digital sensors capture light and convert them into electrical signals. These electrical signals become digital information. Color information is recorded using color filter arrays (CFA) placed over each pixel of the sensor.
16
Color Filter Array
Bayer Pattern Uses twice the amount of green filters because eyes are most sensitive to green wavelength of light.
CYGM Uses colors in both the additive and subtractive system. It uses cyan, yellow, green and magenta. Allows more incident light to be absorbed at the expense of color accuracy.
Input
CYYM Uses the subtractive color system. It uses cyan, yellow and magenta at a ratio of 1:2:1. Used in some Kodak cameras.
RGBW Similar to the Bayer Pattern with the addition of a W, or transparent filter. The W filter absorbs all wavelengths of light making the sensor more sensitive.
17
Chapter 2
Color Filter Arrays are an arrangement of color filters on a grid. The most common arrangement is the Bayer Pattern containing red, green, and blue filters at a ratio of 1:2:1. Green is the most prevalent because it is said to be the most important color to us as our eyes are most sensitive to green wavelengths of light. Here we see multiple CFA patterns.
RGBW An RGBW CFA is an arrangement of 1 Green: 1 Red: 1 Blue: 1 Transparent (W) filter. The new Transparent filter is sensitive to all wavelengths of light therefore, collecting a higher proportion of light that hits the sensor and enhancing light sensitivity. In 2007, Kodak TrueSense Imaging announced their creation of the RGBW CFA. In the example to the right is their arrangement of filters where the transparent (White) filters replace where the second Green filter is located in the Bayer pattern. The new pattern is arranged similarly so that the existing demosaicing algorithm can still be used to render the images. However, this technology had many drawbacks include being prone to more aliasing when used with existing algorithms, trading good low noise performance for better light sensitivity, and often has clipping in the highlights. In 2012, SONY boasted their new and improved RGBW sensors to be used in smartphones in their press release. They developed an image processor to keep image within clipping boundaries. It was set to launch in 2013, but SONY reverted back to RGB sensor and scrapped their RGBW sensor because it, “was found not to meet certain condition’s of SONY’s image quality standards.”
18
Conventional RGB method
Newly developed ‘RGBW Coding’ method
The Sparse Color Filter pattern can be seen here. P stands for panchromatic. This is a diagram showing how the normal RGB filters provide chrominance to an image and the Panchromatic filters provide the luminance to the image.
Tradeoffs In performance the biggest tradeoff is in color resolution because this pattern spreads out the color samples to ‘fit in’ the P pixels. Another is processing complexity because the algorithm requires more memory and all images can only be processed by host computer (meaning not in real time).
The Sparse CFA technology is already implemented in fields such as security, surveillance, intelligent traffic systems, and medical imaging.
Input
19
Chapter 2
Finally, TRUESENSE Imaging has developed a CFA using the Transparent filter (W) called the Sparse Color Filter pattern. In the example to the right you can see the new pattern. They have developed new image processing algorithms to integrate into current camera software. They claim this sensor has superior light sensitivity that is typically seen in monochrome sensors. It can be utilized in low light situations with fast shutter speeds.
Demosaicing In a color image, since each pixel of the sensor is overlaid with a color filter array (CFA), a demosaicing algorithm or interpolation is needed to create a full color rendition of the image where RGB colors are represented at each pixel. Before this complex algorithm is applied, a mosaicked image is created by the amount of red, green, or blue light read by each pixel behind the color filter array. After the demosaicing algorithm or interpolation is applied, a “normal� image is constructed.
Image before demosaicing
Image corrected with demosaicing algorithm
20
CCD vs CMOS CCD
Blooming occurs when a pixel is exposed for too long and overflows. This can be a problem because this overflow also starts to fill adjacent pixels. This causes overexposed areas of an image. Anti-blooming features are built into higher end CCDs by reducing light sensitivity to light by using some of the pixel area for a drain structure.
CMOS CMOS or complementary metal oxide semiconductor works in a similar way to CCD. However, there is electrical circuitry next to each pixel (photosite) converting light to voltage, which is then processed by additional circuitry and converted into a digital image.
Input
21
Chapter 2
CCD or charged coupled device works by transferring light (photons) as an electrical charge. Light hits the pixels (photosites) of a sensor and is transformed into an electrical charge.Think of pixels as a bucket and light as rain. The more rain that fills the bucket, the more information is captured. The rain in each pixel is turned into an output signal (voltage) all at once and read as an image.
3 Processing
ImageProcessingPipeline The Image Processing Pipeline is the process an image goes through from capture all the way to the final edited image. A digital image, much like film, goes through numerous steps to reach a final product. This step by step pipeline will help us understand what it takes for an image to become a file we are satisfied with. This ,process involves energy conversion and complicated algorithms.
CFA Interpolation Information from the RAW data are formulated into three images of each channel (red, green, and blue). These three images are demosaiced to produce one full color image.
24
Image Capture Light enters the camera and is focused onto the sensor where the information is recorded.
Neutral Balance Often referred to as white balance, is the manual adjustment of colors in an image. This step is performed to make neutral colors in the image appear neutral.
Analog to Digital Converter The camera sensor (CCD or CMOS) transfers the light (photons) that hits each pixel (photosites) into an electrical charge. The electrical charges are converted into digital numbers and read as an image.
Gamma Correction Gamma correction is used to make colors captured by the digital camera appear as they did to our eye. It is a nonlinear operation used to change the luminance or tristimulus values in order to correct the overall brightness of an image for computer monitors.
Pixel Defect Correction
Open in RAW Processor
The data from the Analog to Digital Converter is collected and put in a ‘RAW’ file format. These include .NEF, .CR2, .DNG.
A RAW processor such as Adobe LightRoom or CaptureOne are programs used to read RAW files and allow for adjustments.
... Camera Profile (sRGB, AdobeRGB, ProPhoto) Depending on the camera, a certain profile is applied to each image to make sure the colors are replicated accurately. Camera profiles are applied between images and printers to allow for output that most closely matches the display.
Processing
Further Corrections and Save as...(TIFF, DNG)
Exposure and Sharpening The best way to adjust exposure is by moving the levels as the maximum, minimum, and median density values between 0 and 255. The histogram should be considered as not to clip out information. Sharpening can be achieved using methods such as unsharp mask in Photoshop.
Many more artistic choices can be made to a preferred aesthetic. This includes overall and local adjustments. Lastly, the image can be saved or compressed into any preferred file format.
25
Chapter 3
Some pixels may be defective because some pixels may respond to light differently than surrounding pixels. Pixel correction averages out neighboring pixels to correct defective ones.
“RAW” Sensor Data Image Capture
4 Output
PPI and DPI Digital Displays and PPI Digital displays are a necessity to the modern photographer. We use these devices to not only view, but also process our digital images. They provide us with a means to view and manipulate our images in a high quality manner. Digital displays such as monitors, scanners, and digital cameras show continuous tone images. This is achieved with the use of many pixels next to each other. PPI or pixels per inch refers to the amount of pixels an image contains. Therefore, the more pixels per inch the better the quality of the image. When there are not enough PPI in an image, gradation happens because the image appears less continuous giving the “pixelated” appearance. Standard monitors display at 72ppi.
inch in order to achieve a picture that appears as a continuous tone image. Tiny dots of ink are laid out in order to create an image. The more dots per inch, the more continuous an image will appear. RIPs or rastor image processor is needed to prepare a continuous tone image for printing. A RIP will convert a vector image’s language, called Postscript (PS), into a raster image that displays the image’s data pixel by pixel. A RIP can be present in the computer program, printer itself, or as a separate unit.
Digital Image
Printers, DPI, and RIPs Printing allows our images to become physical objects that we can easily share, possess, and touch. When discussing printing, the cheapest and most popular methods concern DPI or dots per inch. Digital printers include halftone printers, inkjets, and laserjet. These are all binary devices meaning they use DPI or dots per
28
Zoomed 3200% can see individual pixels
Halftoning and LPI Halftoning Halftoning occurs when printing using dots. The use of DPI creates the appearance of a continuous tone image. A successful halftone image has to overcome the human eye’s resolving power to produce, what appears to be, a continuous tone image. Different tones are created by the physical space between dots. Halftone printing consists of CMYK dots.
S
Output
LPI or lines per inch is a measurement of printing resolution. It measures how close the lines in a halftone grid are. The higher the LPI, the greater the detail and sharpness of the image.
600 DPI
1200 DPI
29
Chapter 4
LPI
Printer Technologies Analog Analog printing is the process in which manually prepared screens or plates are used for printing onto a type of material. Gravure is used for commercial printing. It involves an image being engraved onto a copper plate wrapped around a cylinder with an impression roller against it. The gravure cylinder rotates through an ink fountain while a blade squeegees off excess ink. A roll of paper is sandwiched between and ink is applied. Printing is fast, produces high quality images, and is good for long runs of printing. On the downside, it takes time and is expensive to engrave new plates, and is not good for short runs of printing.
Gravure printing process
Digital Digital Printing is the process in which computerized controlled ink droplets are ejected onto a type of material. Laser printing is used most in office environments. They have a high throughput, produce halftone images, are low cost per print, have high text quality, and their light and water fastness is better than inks. On the downside they have a high cost per unit and have produce poor photo quality prints.
30
An example of an office laser printer
Thermal inkjet where heat produces air bubbles
On the high-end of inkjet printing is continuous tone printing. It can produce large size prints (30’’ x 40’’) and has a high initial cost of $30,000. IRIS prints are closely associated to this technology, but are being phased out by large format inkjet printers. Dye-sublimation printing produces the highest quality photographic prints. They produce continuous tone images and the way in which dye is absorbed into paper is superior. The same amount of dye is used for each print regardless of picture content. On the downside there is a high cost per print, requires special paper, have slow printing speeds, and restricted output sizes. Dye-sublimation printer process
Output
31
Chapter 4
Piezoelectric Inkjet where charged crystals expand
Inkjet is a popular type of printing and now drop on-demand is the most prevalent. They are relatively inexpensive, use a plain paper output, uses the halftone method, and produce prints relatively quickly. On the downside the water fastness of some inks are still a problem. The two main types of drop on-demand printers are thermal and piezoelectric inkjets. Thermal ink jets uses heat to form bubbles that eject ink onto paper. Piezoelectric technology ejects ink through a crystal. Positive, negative, or no voltage controls the shapes, volume, and speed of ink droplets.
32
Credits Film
http://www.hatiandskoll.com/wp-content/uploads/2013/02/PhotographicFilm.png http://scanyourentirelife.com/wp-content/uploads/2011/12/scan_negative_vs_print-negative_and_print_side_by_side.jpg
Digital
http://upload.wikimedia.org/wikipedia/commons/1/1c/Bayer_pattern_on_sensor_profile.svg http://cpn.canon-europe.com/files/education/infobank/capturing_the_image/cmos.jpg
CFA
http://en.wikipedia.org/wiki/Color_filter_array
CCD/CMOS
http://www.siliconimaging.com/ARTICLES/CMOS%20PRIMER/image015.jpg http://meroli.web.cern.ch/meroli/lecture_spatialResolution/clip_image002.jpg
Image Pipeline
http://farm6.staticflickr.com/5208/cameras/72157624637608383_model_huge_360fd19978.jpg http://d1ax9dx3gero0.cloudfront.net/wp-content/uploads/2013/11/lightroom_5_logo_large_verge_medium_landscape.jpg http://www.ephotozine.com/articles/capture-one-pro-7-offer-phase-one--capture-one--offer--software-22133/images/phaseone-logo.gif http://4.bp.blogspot.com/-l4chW15qSf8/TbstFyIxn9I/AAAAAAAAAEE/bO8oj0V39dk/s1600/003.jpg http://www.mintviz.com/workshop/wp-content/uploads/2011/05/gamma-03.jpg http://upload.wikimedia.org/wikipedia/commons/9/9d/Munsell_1929_color_solid_transparent.png http://godigitalslr.com/wp-content/uploads/2012/01/MeterScale-typical.jpg
Halftone
http://cdn.photo-kako.com/sample/halftone.jpg
LPI
http://www.sharp-world.com/products/img/copier/products/mx_c381_c311_c380_c310/1200dpi_printing.jpg
Printing
http://www.automation.siemens.com/mcms/mc-solutions/en/mechanical-engineering/printing-machines/gravure-printing-machine/PublishingImages/zoom_gravure-printing.jpg
33