Emily Rollin

Imaging Systems Foundations

Written by Emily Rollin with help from Nitin Sampat

Capture

All images by Emily Rollin unless otherwise noted

Process

Produced during the 2013 fall semester at RIT for the Imaging Systems minor

Output

This is a summary of knowlege from a semesters study

Imaging Systems

Table of Contents 4 11 14 16 17 20 22 23 24 28 29 30 31

Fundamentals Image Resolution File Formats

Capture

Camera Sensors Demosaicing CCD and CMOS

Process

Image Pipeline Image Stabilization Sharpening LAB

Output

Displays Printing DPI and PPI Half-toning

Foundations

The foundations of an image include its spacial, tonal and spectral resolution. The foundations of a moving image also include temporal resolution. This chapter briefly introduces file formats and the sizing and compression of an image as well as image resolution.

Image Resolution Resolution relates to the amount of detail in an image. There are four kinds of resolution: spatial, spectral, and tonal resolution. Multiplying the resolutions gives the file size of the image. To find the file size of a moving image, you must also factor in temporal resolution. black and white image size = spatial x tonal color image size = spatial x tonal x spectral moving image size = spatial x tonal x spectral x temporal

8 | Foundations

Figure 1: Original image- The white box indicates the selection of the image enlarged 200% in Figures 2-4

Spatial Resolution

Figure 2: Bilinear interpolation

Figure 3: Bicubic interpolation

Spatial Resolution determines the sharpness of lines and details in an image. The higher the special resolution, the finer the detail is in the image. Spatial resolution is determined by the number of independent pixels in an amount of space.

In order to enlarge a digital image, software like Adobe PhotoShop has to estimate new pixels based on the existing pixels. This process is called interpolation.

Examples of different methods of interpolation are shown in figures 2-4.

Bilinear Interpolation, demonstrated in Figure 2, samples the 2x2 area surrounding a missing pixel and averages those pixels together to create the interpolated pixel.

Bicubic Interpolation, demonstrated in Figure 3, samples the 4x4 area, 16 pixels, surrounding a missing pixel. It gives more weight to pixels closest to the missing one in the average, generally producing a sharper image than other methods of interpolation.

Figure 4: Nearest neighbor interpolation

Figure 5: The white squares represent pixels that are missing information after an image has been enlarged. Using Nearest Neighbor, the missing pixels have been created by averaging the closest pixel to the left.

Nearest Neighbor Interpolation only considers the pixel most near to the missing data. Because nesrest neighbor interpolation is duplicating the nearest pixel, the final image apears to have larger pixels than it actually does. The edges in an image with this type of interpolation appear jagged as shown in figure 4.

9

Tonal Resolution Tonal Resolution of an image defines the number of levels between the darkest dark and lightest light. The more levels recorded in an image, the smoother it appears. If an image has only a few levels, our eyes are able to easily distinguish the levels individually. This is demonstrated in figure 1, at 4 levels and 8 levels. At 256 levels, 8 bits/channel, the limit of the human eye is reached.

A bit is the fundamental unit of information. Bit depth is determined by the amount of bits needed to tell the color of a single pixel. Computer monitors display at 8 bits per channel (RGB) and output 256 levels.

256 levels

64 levels

16 levels

8 levels

4 levels 10 | Foundations

Figure 7: This image demonstrates posterization at different levels. Posterization is most noticable in places where the tones are continuous, the sky being a good example.

Histograms help to represent the distribution of tones in an image by plotting each pixel according to its level of brightness. A histogram tells if any information in an image is lost, the tonal quality of the image, as well as the dynamic range (contrast).

8 bits/channel

The left side of a histogram represents the darkest part of an image and the right side represents brightness. The histogram in Figure 10 shows that the image came close to losing information in the highlights as the pixels are plotted precariously close to the right edge.

6 bits/channel

Max pixel count

4 bits/channel

0

Levels

256

2.83 bits/channel Figure 10: Histogram of Figure 7

2 bits/channel Figure 8: These gradients help to demonstrate posterization.

11

Spectral Resolution Spectral Resolution, or color resolution, represents the color space used in an image. The colors of an image are created through either the Additive Color System (RGB), Subtractive Color System (CMYK), or another color space.

Additive Color Color is created by mixing the light of two or more additive primary colors: red, green, and blue. When all three colors are combined, white is created and when all three colors are absent black is created. An image in RGB color space can be divided into the three monochrome channels, red, green and blue. When the channels are combined they yield neutral colors.

Additive color can be seen on computers and television monitors that use light to display an image.

Red

12 | Foundations

Blue

Green

Figure 11: When you apply a color filter to the front of a camera, all light of that color is absorbed. An example of what the captured image might look like is portrayed above.

Subtractive Color By mixing dyes, inks and pigments, a wider array of colors can be represented in print. The primaries of this color space are cyan, magenta, and yellow. When all three colors are combined, black is created which is the opposite of RGB color space.

Red - (green + blue) = Cyan Green - (red + blue) = Yellow Blue - (red + green) = Magenta White - (red + green + blue) = Black

Cyan

Magenta Magenta

Yellow

13

Temporal Resolution Temporal resolution is the refresh rate of the monitor when displaying a moving image. The higher the refresh rate, the better the quality. Time resolution for today’s TV and movies is between 24 and 48 frames per second.

Figure 12: Pictured is a frame extracted from an HD movie shot at 24 frames per second 14 | Foundations

File Formats File Format is the way the data of an image is stored. Each format is designed specifically for certain kinds of data. For example: TIFF (Tagged Image File Format) -8 bits/channel (24 bit color) -Lossless image compression -RGB Color Space

Compression Figure 13: Maximum JPEG compression - 128 kilobyte file size

The compression of an image reduces information that isn’t needed or is redundant, in turn it shrinking the file size.

Lossless compression keeps the integrity of an image intact and is the preferred method for archiving. Lossy compression minimizes the amount of data retained in the file. As an image is compressed, visual integrity is compromised. In Figures 13-15 one can see how image quality is sacrificed to get a smaller

Figure 14: Medium JPEG compression - 39.8 kilobyte file size

Figure 16: (below) This screen shot shows the JPEG compression options offered when saving an image in PhotoShop

Figure 15: Minimum JPEG compression - 27 kilobyte file size 15

Capture

When photographing, an image must be captured on a sensor. This chapter explores film and digital sensors, including CCD and CMOS and touches on Demosaicing and color filter arrays.

Camera Sensors The sensor of a digital camera and the film used in an analog cameras both have light sensitive qualities that record information when exposed.

Film Film is made of several layers. The bottom layer is a gelatin emulsion containing silver-halide crystals, which are light sensitive. Color film generally has a blue, red, and green layer atop the light sensitive layer. The color from the layers is absorbed and recorded in the latent image, which is invisible until the film is developed.

Figure 17: Image created on film

Digital Sensor Digital sensors work by capturing light and turning it into an electronic charge. The electronic charges are converted to volts and then to digital information. A grid of color filters is placed over the sensor so that color information is recorded as well.

18 | Capture

Figure 18: Image created on a digital sensor

CCD and CMOS Charged Coupling Devices (CCD) and Complementary Metal Oxide Semiconductors (CMOS) are both common image sensors used in digital imaging. These sensors are what change light from the scene being photographed into electronic signals. There’s much debate around which sensors present the most advantages.

Charged Coupling Device CCD sensors are found in high consumer point-and-shoot cameras such as the Canon PowerShot SX500 and the Sony CyberShot DSC-H200 show at right.

A CCD sensor has many photo sensors and when light strikes the chip, it’s held in the sensors until converted to voltage one by one. In a CMOS sensor, circuitry next to a photo sensor converts light to voltage.

Neither process has proved better than the other.

CMOS sensors have been recorded as faster and they can be more cheaply manufactured than CCD sensors but when it comes to quality results, they are equals.

Complimentary Metal Oxide Smiconductors CMOS sensors can be found in professional DSLRs like the Canon 5D Mark III and the Nikon D800 shown at right.

19

Color Filter Array Color filter arrays are filters placed over each pixel on a digital sensor. As light from the scene is reflected onto the sensor, the cfa filters that light in one color and later, the image is demosiaced so the color information can be corrected to neutrals.

The Bayer Pattern is the most commonly used CFA pattern. It is also called the RGBG pattern because it features twice as many green receptors as red and blue which is comparable to how the human eye sees color.

Bayer Pattern

RGBE

CYGM

RGBW1

CYGM

Bayer RGBW

Many Modifications of the Bayer pattern have been created in attempt to enhance color images or to add a unique factor to the camera.

Incoming light

20 | Capture

Demosaicing Demosaicing, like spatial interpolation (see page 9) is the process of creating information missing from an image. In this case, color information is missing. The color filter array over the camera sensor tells us how much red, green, or blue light was read by that particular pixel, creating a mosaicked image that would look something like Figure 19. Demosaicing is an algorithm that fills in the missing color information, creating a natural looking image.

The process of demosaicing occurs in camera before we ever see the image, but some cameras can provide the photographer a pre-demosaic file, allowing them to demosaic in post editing.

Figure 19: Example of what an image prior to demosaicing would look like

Figure 18: This image has been corrected using demosaic algorithms

21

Process

When an image is captured with a digital camera, it must undergo the image pipeline before it’s in a form easily deciphered by the human eye.

The photographer may also make artistic and aesthetic corrections before the image is published.

Image Pipeline The image pipeline is the process a digital image goes through between the moment it’s captured and when we are able to see it. Like film, the image must be ‘developed’ but instead of chemicals and rinses, digital photography involved algorithms and energy conversion.

The image pipeline helps us understand how the light from the captured scene becomes an image we can edit and manipulate until it suits our needs.

1

3

Image Capture

Digital Image Sensor

Pixel Defect Correction

The camera shutter opens allowing light to enter the camera where it is then focused on the sensor and recorded.

The camera sensor (CCD or CMOS) has thousands of pixels, each holding an electrical charge from the incoming light.

A sensor may have a number of pixels that respond to light in irregular ways. Pixel correction samples neighboring pixels to correct the defective ones.

8

24 | Process

2

9

10

Raw Processor

CFA Interpolation

Neutral Balance

A RAW processor like Adobe LightRoom and CaptureOne is used to decipher the data files to create color and make adjustments.

The information from the color filter array is translated into three monochrome images (red, green, and blue) and each is demosaiced (see page 20).

The colors of the image are adjusted so that there is no color cast and the colors appear similar to what they were in the original scene.

4

5

6

7

Pixel response nonuniformity correction is the noise in an image that occurs when pixels output values that aren’t uniform, creating noise. It is almost impossible to eliminate all noise.

The electrical charge recorded at each pixel on the sensor is converted to volts so that it can be interpreted by the ADC.

The ADC (analog to digital converter) converts voltage from each pixel into a digital number to be read by the computer.

The unedited data from the ADC is collected and put into a ‘RAW’ file format (.CRW, .NEF, .DNG)

11

12

13

14

A color profile is applied to the monitor and printer so the printed output looks the same as the image on the display.

From this point the image can be edited as desired until it reaches the intended artistic aesthetic.

PRNU

Gamma Correction

Humans are able to see at a certain bandwidth. Prior to gamma correction, the image will have too much bandwidth to the highlights and is too bright for us to see.

Analog Signal

Camera Profile

Each digital camera has a profile applied to every image to ensure the colors from a scene are replicated properly.

Camera profiles can be customized for specific outcomes.

Analog/Digital Converter

Calibration

Raw Data

Further Corrections

25

Image Stabilization Image stabilization is a technique created to help reduce the amount of blur in images taken in low light or with long focal length lenses and slow shutter speeds.

In-lens stabilization uses two gyroscopes, little machines that record the vibrations created by humans when hand holding the camera (Figure 23). A small computer in the lens then uses that information to make adjustments to the lens group (Figure 24) keeping the image on the focal plane.

Benefits of having the stabilization in the lens as opposed to in the camera are that the autofocus system is able to work more easily if the lens is stabilized. Also, having the image stabilized in the lens means it’s also stable in the viewfinder, which makes framing the image easier for the photographer.

Figure 21: Image stabilization off

Figure 22: Image stabilization on

Figure 23: The arrows show which movements the gyroscopes are able to record

Figure 24: This image group demonstrates how in-lens stabilization works

Canon’s Optical Image Stabilizer technology offers in-lens stabilization systems for long focal length lenses, macro lenses, and digital video cameras. Even their consumer point and shoot cameras have OIS systems.

26 | Process

Sharpening Image sharpening is a tool used to exaggerate texture and clarify focus. There are many applications that can be used to sharpen an image but if you aren’t careful, sharpening can have degrading affects, like sharpened noise and sharpening halos.

The unsharp mask is a technique that works by blurring an image then subtracting the origintal image from the blurred image, producing a mask like the one pictured above.

Blurred portions of an image have a low frequency and sharp aspects have a high frequency. When sharpening an image you’ve effectively decreased the amount of low frequencies.

The most commonly used method of sharpening, even if you don’t know it’s being used, is the unsharp mask, which was a technique developed in the days of film.

The next step in sharpening an image using unsharp mask is adding the mask back to the original iamge. The image is now appears less blurry than the original. This is what the final sharpened image looks like.

Figure 25: Original image 27

LAB Color Lab color space is a 3D model arranged on three axises (see top left image). Lab is designed to mimic human vision, unlike RGB or CMYK. This color space has a ‘lightness’ component, also to match human perception. Lab color is based off of the Munsell color system pictured on the top right of this page.

L in LAB represents lightness or density, 0 being black and 100 being white or lightness.

In a and b channels, values range from -128 to 127. Positive a values yield red or magenta, positive b values yield yellow.

Negative a values yield green, negative b values yield blue.

For example, below is a Pantone color chip.

Its Lab color value is: L: 53 a: -69 b: 2

Figure 26: Above is a 2D representation of LAB color space 28 | Process

It can be hard to pump up the colors in a photograph without making it look unnatural. In RGB mode, changing your colors in curves will also produce changes in brightness and exposure. To fix this, we can easily switch to LAB Mode by selecting it from the Image drop down menu.

If you pull up a curves adjustment layer in LAB, you’ll see that your red, green, and blue channels have been replaced with Lightness, A, and B (see example iamges below). Your A and B Channels ONLY affect the colors in your image, so you’re free to push them as much as you desire without messing up your exposure.

-PHLearn.com

Original image

Image edited in LAB color space

29

Output

The output of an image is seeing it on a display like a television, computer monitor, tablet or smartphone or in a physical form like a print. This chapter teaches the difference in output on a display and a printer as well as printing techniques.

Displays Liquid crystal display is a flat panel display, electronic visual display, or video display that uses the light modulating properties of liquid crystals. They are more energy efficient than cathode ray tube (CTR) displays.

Characterization Also known as profiling, characterization is recording the way a display or printer reproduces color, as each will do so in a slightly different way. Ultimately, you want to produce an image in print that looks the same as how it looks on the display. This process is called calibration. By adding a profile to the computer display and the printer, matching color is possible.

Devices like the Xrite ColorMunki create a profile from a color sheet produced by a printer and then apply it to the display to create image color results as close as possible.

32 | Output

Printing Web offset printing is a technique utilized by mass printing industries. The type or image being printed is transferred to a metal plate, then to a sheet of rubber, which is coated with ink and used to transfer the image to the paper. Paper is fed through the machine on giant rolls and the machines print continuously.

Inkjet printing prints a digital image by dropping ink onto the desired surface, currently the most commonly utilized printer type. Continuous inkjet printers (CIJ) uses pressure to propel ink through the nozzle into the paper. Drop on demand printers (DOD) heats the ink until it bubbles and drops onto the print surface.

Laser printing creates a print from digital image by passing a laser beam over a charged drum, which collects charged toner and transfers the image to paper. The toner is then heated, fixing the image to the paper. This process is very rapid and so laser printers are often found in offices, utilized for printing documents.

33

DPI, PPI, LPI A dot is the smallest binary element that a device can generate. DPI or dots per pixel is the amount of ‘dots’ a device can print or display in one inch. ‘Dots per inch’ is a term applicable to inkjet and laser printers, which print with dots.

A pixel is the smallest element in an image that can display levels of gray. PPI or pixels per inch is the number of ‘pixels’ a device can display or print in an inch. LPI or lines per inch is a term originating from analog printing which is effectively the same as PPI. ‘Pixels per inch’ is a term used in reference to scanners, digital cameras, monitors and continuous tone printers.

Above is a sampling of the original image of Hanna with her gun magnified 1200% in Adobe PhotoShop. Each seperate square is one pixel.

34 | Output

Above is an example of what printed ‘dots’ look like.

Half-Toning Half-toning is a technique that translates a continuous tone image into dots varying in size, shape, spacing and color. Halftoning an image produces an optical illusion, fooling the eye into seeing a smooth image that is actually created by dots. Because the ink is semi-opaque, cyan, magenta, yellow and black dots can be layered atop each other to create a halftone image in full color (see Figure 26).

Figure 26: Demonstrates color half-toning

Image that has been half-toned.

35

Credits

RGB Color CMYK Color Film Layer Bayer Sensor CFA filter CMOS &CCD CMOS Camera Nikon Df White Bal. Contact Sheet CFA pattern Figure 23 Figure 24 Red Camera Sharpen Halo Canon OIS Dots per inch LCD Calibration ColorMunki LAB color space L*a*b* Munsell

Wikipedia (http://en.wikipedia.org/wiki/RGB_color_space) Wikipedia (http://en.wikipedia.org/wiki/CMYK_color_model) Hati & Skoll (http://www.hatiandskoll.com/author/admin/page/25/) Wikipedia (http://en.wikipedia.org/wiki/Bayer_filter) Wikipedia (http://en.wikipedia.org/wiki/Color_filter_array) Teledyne (http://www.teledynedalsa.com/imaging/knowledge-center/appnotes/ccd-vs-cmos/) SnapSort (http://snapsort.com/explore/best-digital-cameras/24-months-recent-cmos-sensor-type) NikonRumors.com (http://nikonrumors.com/2013/11/04/this-is-the-nikon-df.aspx/) PhotoWalkPro (http://www.revellphotography.com/blog/2009/12/when-should-you-white-balance/) Caroline’s Blog (http://carolinephotography5.blogspot.com/2010/09/contact-sheet-1.html) Wikipedia (http://en.wikipedia.org/wiki/Demosaicing) Canon.com (http://web.canon.jp/imaging/lens/technology/index3.html) Canon (http://www.usa.canon.com/cusa/consumer/standard_display/Lens_Advantage_IS) http://www.geeky-gadgets.com/red-scarlet-23-camera-specifcations-and-pricing-revealed-01-12-2009/ http://www.designsbysusan.com/t_painting.html http://de.wikipedia.org/wiki/Datei:Canon_Optical_Image_Stabilizer_Logo.svg http://99designs.com/designer-blog/2013/02/26/ppi-vs-dpi-whats-the-difference/ http://en.wikipedia.org/wiki/Liquid-crystal_display http://www.argyllcms.com/doc/calvschar.html http://xritephoto.com/ph_product_overview.aspx?id=1115 http://www.metropostcard.com/guidecolor.html http://dba.med.sc.edu/price/irf/Adobe_tg/models/cielab.html http://www.colortheoryresearchmunsell.blogspot.com/

Imaging Systems The Imaging Systems minor offers students an introduction to the business and technology of photographic imaging services. Courses cover digital imaging capture systems, professional practices, output technologies, color management, and imaging workflows.

The minor provides the foundation students need to pursue opportunities in business management, photo-lab operations, technical support, digital imaging technology, and sales for photo and imaging manufacturers.