Volume 2, Spl. Issue 2 (2015)
e-ISSN: 1694-2310 | p-ISSN: 1694-2426
An efficient global registration method for multi-exposure images Harbinder Singh, Vinay Bhatia Electronics and Communication Engineering Department, Baddi University of Emerging Sciences and Technology Baddi, Solan 173215, India e-mail: harbinder.ece@baddiuniv.ac.in, vinay.bhatia@baddiuniv.ac.in Abstract— In this paper we present a method for eliminating global misalignments between a sequence of images captured at different exposures with horizontally moving camera. The proposed method utilizes template matching based on normalized correlation to find the correspondences between different input images. The best known correspondence is used to transform a set of images to a single coordinate system and eliminate any global misalignment (including translation). The proposed registration technique works well for under-and over-exposed images usually required for High Dynamic Range (HDR) image generation and construction of HDR panoramic image. Any camera movement results blurry HDR image. The results show that the image registration accuracy of our method is same when images captured at large exposure difference. Uses of different 2-D geometric coordinate transformations in consecutive images are also discussed. Keywords — Image registration, Multi-exposure, Template matching, High Dynamic Range Image
I.
INTRODUCTION
Image registration of two-dimensional images is a fundamental process in digital image processing to estimate the underlying correspondence between two or more images. Image registration process compensates misalignments caused by any movement of camera (global motion) or dynamic object in a scene (local motion). The variety of applications worthwhile: HDR image generation, image recognition, image mosaics, artifact reproduction, image fusion and many others such as augmented reality in graphics and map building in robotics. Image registration methods can be divided into feature and area based methods. Similarity measures are used in image registration process to determine correspondence between all points in two or more 2-D images [1] and [2]. In the case of HDR image generation HDR radiance maps are recovered from the differently exposed images [3]. Most techniques assume that the input images captured at different exposure values with a fixed camera. One of the main requirement of compositing HDR image from multiple exposures using the technique proposed by Debvec and Malik [3] is that the camera be absolutely still during image
291
capturing process. Unfortunately, even tripod mounting will sometimes allow slight shift in the camera. The proposed method is dedicated to problem of global registration (including translation) of images captured by hand- held camera at varying exposure values. Method is based on template matching of reference image with windows of same size in other images, which are captured for different EV. The similarity metric used here is normalized correlation to identify window that is most similar to the template of reference image. In the proposed image alignment technique normalized correlation is used for the estimation of translation parameters. Different similarity metrics have been developed and the selection of similarity metric is depends on types of images provided for alignment. A review of various similarity metrics used in image alignment is given in [1]. Our method is fully automatic and compensates global translation of multiexposure images. A review of previous work related to image registration and HDR image generation is given in Section 2. In Section 3, we describe our method for image alignment, and measure normalized correlation of several images captured at different exposure values. Various geometric transformations to modify the spatial relationships between pixels in input and output images are also discussed in Section 3. In Section 4, we discuss the implementation of algorithm, and finally, Section 5 concludes the paper. II.
RELATED WORK
In recent years many algorithms have been proposed to capture HDR images from bracketed images (standard LDR images) [3], [4], [5], and [6].The researchers suggest using tripod to avoid any global movement. The problem of image alignment of multi-exposure images to capture HDR image was proposed in [7] and [8].They were employing conversion of input photographs in to percentile threshold bitmaps for image alignment. We have found out that few conventional approaches dicussed in [1] and [2] to image alignment usually fail when applied to images with variable exposures. As shown in Fig. 1 edge detection filters are exposure dependent, where edges appear and disappear at different exposure levels. Therefore
BUEST, Baddi
RIEECE -2015
Volume 2, Spl. Issue 2 (2015)
e-ISSN: 1694-2310 | p-ISSN: 1694-2426
where N is the relative aperture and t is the exposure time (shutter speed) in seconds. Image data set used in our algorithm is captured by the camera at three different exposure settings given in Table I. We describe the algorithm on N 8bit grayscale images. These images are first converted into Y CbCr format. The relationship between RGB and YCbCr is given by: (a) Edges of normally-exposed image captured at exposure value (EV=0).
Y = 0.3 R + 0.59G + 0.11B
(b) Edges of over-exposed image captured at exposure value (EV=2).
(2)
Normally exposed (i.e. value of EV for this image is zero) is selected as the reference image out of all N images captured. The output of the algorithm is a series of (N-1) 8-bit images. Size of template cropped from normally-exposed (i.e. EV=0) is 51 × 51. After capturing and choosing a reference image we move to the second step which is similarity measurement and discussed in the forthcoming section. B. Similarity Measures
(c) Edges of under-exposed image captured at exposure value (EV=-2) Figure 1. Edge detection of images captured at different exposure settings using Canny operator
conventional approaches (Exposure dependent) are ill-suited for alignment of images captured at different exposure values. A. Tomaszewska et al. [9] presented fully automatic method for eliminating misalignment between a sequence of images captured by hand-held camera at different exposures. This technique is based on SIFT method to search the keypoints (or feature points) in consecutive photographs. III.
PROPOSED ALGORITHM
This section presents detailed description of our algorithm for registration of globally translated images captured at multiple exposures. The proposed registration algorithm consists of three stages: the second and the third stage are applied in the spatial domain for image alignment and the first stage is the capturing of images at variable exposures settings. A. Multiple exposure Image data set collections Multiple exposures collection also called exposure bracketing is the process to capture photographs at different exposure values (EV). Negative exposure (under-exposed) are used to capture brightly lit details (highlights) and positive exposure (over-exposed) are used to capture dark details (shadows). In photography, EV denotes combination of a camera’s shutter speed and relative aperture. Exposure value [10] is a base-2 logarithmic scale defined as: N2 EV = log 2 (1) t
292
Similarity measure given in (3) calculates normalized correlation between template of reference image and at each possible location in other images captured at variable exposures. The maximum value or distinct peak thus obtained at position, where template of reference image and other image is exactly matched. To identify the global translation of distinct peak in reference and other images, Normalized correlation is calculated as follows:
λ ( x, y ) =
∑
∑
s ,t
s ,t
[ w( s, t ) − w][ f ( x + s, y + t ) − f xy ]
[ w( s, t ) − w]2 ∑ s ,t [ f ( x + s, y + t ) − f xy ]2
(3)
where w=
1 M
f x, y =
∑ w( s, t )
(4)
s ,t
1 M
∑ t ( x + s, y + t )
(5)
s ,t
In (3), w is the average value of the elements in template, f x, y is the average value of the image in the region where template ( w ) and image ( f ) overlap. M is the number of elements in template and window of same size in other images. Since λ represents normalized values therefore it ranges from -1 to 1. Fig. 2 shows normalized correlation between template of reference image and other images. It is clear from Fig. 2 that as template and window of same size in image become similar the value of λ(x, y) becomes large. Therefore we can conclude that the point where template and window matches exactly λ(x, y) has maximum value. The coordinates of the maximum values of λ denoted by λ m ax give the estimates for the translation parameters. After the estimation of maximum values (distinct peaks) of λ(x, y) in all input images, correct translation parameters
BUEST, Baddi
RIEECE -2015
Volume 2, Spl. Issue 2 (2015)
e-ISSN: 1694-2310 | p-ISSN: 1694-2426
on image sizes and the size of template. We have chosen 51×51 template of normally-exposed for image alignment. Once the translation parameters are calculated, spatial relationships between pixels in reference image and other images are modified and discussed in next section. C. Geometric Coordinate Transformations Geometric transformation is the process to modifying spatial relationships between pixels in input and output image. Coordinate transformation of input pixels to new positions are defined in terms of geometric coordinate transformation as follows: g ( w, z ) = f [T −1 ( x, y )] (8)
(a) Correlation between template of reference and normally exposed image
In (8), f(x,y) denotes an image in the input space, g(w,z) image in output space after transformation, and T −1 {•} is a mapping function.
(b) Correlation between template of reference and underexposed image
Once distinct peak (correspondences) in all images have been identified using (3), We can now describe how coordinates are mapped between the input and output pixels. Mapping function should transform pixels of sensed image to new location according to the geometric deformation required. In our case similarity transform approximated by (19) has been used. Similarity transform preserves angles between lines and changes all distances in the same ratio to preserve shape. The simplest 2-D (planar) motions are:
(c) Correlation between template of reference and underexposed image
Identity: 2D identity can be written as:
x=w y=z
Figure 2. Normalized correlation between template of reference and images captured at variable exposures
(9) (10)
Translation: This transform translate input vector spaces to new location and can be written as:
Figure 3. Template of reference image (normally–exposed image) used in translational shift estimation with size of 51×51.
denoted by (δ x , δ y ) and are calculated using (6) and (7). The differences between x-coordinates and y-coordinates of distinct peaks estimated across the reference image and one of the (N-1) image is calculated as follows:
δ x = λmax1 ( x ) − λmax 2 ( x ) δ y = λmax1 ( y ) − λmax 2 ( y ) where
λmax1 ( x)
and
(6)
(7)
λmax 2 ( x)
x = w +δx (11) y = w+δy (12) where δ x and δ y are translation factors for horizontal and vertical shift respectively.
Rotation: 2D rotation transform can be approximated as: x = w cos θ − z sin θ y = w cos θ + z cos θ Rotation + Translation: x = w cos θ − z sin θ + δ x y = w cos θ + z cos θ + δ y
are the values of x-
where
coordinate of distinct peak in reference image and image to
angle.
be registered respectively, and
λmax1 ( y )
and
λmax 2 ( y ) are
values of y-coordinates of distinct peak in reference image and image to be registered respectively. Computational time of distinct peak and translation parameter calculation using normalized correlation depends
293
(13) (14)
δx
and
(15) (16)
δ y are the translation factors, and θ is rotation
Scaled Rotation: Also called similarity transform, which preserves angles between lines. x = S x w cos θ − z sin θ (17) y = w cos θ + S y z cos θ (18)
BUEST, Baddi
RIEECE -2015
Volume 2, Spl. Issue 2 (2015)
e-ISSN: 1694-2310 | p-ISSN: 1694-2426
IV. RESULTS AND DISCUSSION We have implemented an image registration method that uses template matching to determine best correspondences in the input images captured at different exposure settings. We performed image registration experiments on images taken with hand–held digital camera (canon 10D with Sigma 18–125
(a) Template position in normally exposed image.
(a) Aligned normally and under-exposed images are superimposed on each other (background is normally–exposed and foreground is under–exposed).
(b) Template position in under-exposed image
(b) Aligned normally and over-exposed images are superimposed on each other background is normally-exposed and foreground is over-exposed).
(c) Template position in over-exposed image Figure 4. Experimental results of similarity measure for template matched in normal, under, and over exposed images. Blue asterisk on image shows point where template of reference image is exactly matched in all images.
where Sx and Sy are scale factors, and θ is rotation angle. Affine: An affine transformation can be written as: S cos θ S sin θ 0 T = − S sin θ S cos θ 0 δ 1 δy x
(19)
Affine transform in (19) includes rotation, scaling, translation, and reflection. It is an important mapping similarity transform used in image registration. Projective: It is useful geometric transformation, which also known as a perspective or homography transform. It can be written as: a11 a12 a13 [ x ' y ' h ] = [ w z 1] a21 a22 a23 (20) b b 1 1 2 Unlike for affine transformation, where a13 and a23 are non zero and auxiliary coordinate denoted by h. In (20) x = x '/ h and y = y '/ h . In projective transformation parallel lines do not stay parallel.
294
(c) Aligned under-exposed and over-exposed images are superimposed on each other (background is under-exposed and foreground is over-exposed).
Figure 5. Experimental results of similarity measure for template matched in normal, under, and over exposed images. Blue asterisk on image shows point where template of reference image is exactly matched in all images.
mm F3.5-5.6 DC lens) and encompassed possible horizontal and vertical translation. We have assumed that there is no illumination change during image capturing process. In all the experiments, we have adopted similarity measure given in (3) to determine correspondence points. Normally–exposed image is selected as the reference image and the size of the template cropped from reference image is 51×51 that is shown in Fig. 3. In the similarity measure test, the template image in Fig. 3 was shifted horizontally and vertically to find the correspondences in search image at each possible location. As shown in Fig. 4 distinct peaks are identified in all input images, where template is exactly matched in the search images. The differences between the distinct peaks calculated by (6) and (7) in the
BUEST, Baddi
RIEECE -2015
Volume 2, Spl. Issue 2 (2015)
e-ISSN: 1694-2310 | p-ISSN: 1694-2426
reference image and one of the (N-1) image are given in Table II. Difference between the locations of distinct peaks determined with (3), (6), and (7) are used for spatial transformation that are listed in Table II. To check the accuracy of proposed registration method two images are superimposed on each other based on determined corresponding errors (translation parameters) in Table II. The superimposed aligned images are illustrated in Fig. 5. As shown in Fig. 6, the fusion of two unaligned images produce ghost image. Computation time for proposed image alignment on images of size 534×364 is about 10 seconds on i5-460MPC.
V. CONCLUSIONS AND FUTURE WORKAND DISCUSSION In this paper we have presented a technique for registering the images captured at variable exposure differ by translation. This algorithm is useful for images taken from same sensor and which are misaligned by horizontal and vertical translation. The method gives reliable results even for images taken with hand-held camera at large exposure difference. But limitation of this algorithm is inability to register the dissimilar images, having different information and self-similar, having same information repeating in images itself. Finding the algorithm for HDR image and HDR panoramic image generation from multi-exposure images after alignment is the subject of future research. REFERENCES [1]
Figure 6. Fusion of unaligned normally and under exposed images.
Table I PROPERTIES OF MULTI-EXPOSURE IMAGE DATA SET.
Image type Normally-exposed Under-exposed Over-exposed Template image
Exposure value 0-EV -2-EV 2-EV 0-EV
Format JPEG JPEG JPEG JPEG
Resolution 534×364 534×364 534×364 51×51
Table II CO R R E S P O N D I N G E R RO R S B E T W E E N L O C AT I O N S O F D I S T I N CT PEAK A FTER D ETER MIN IN G SIMILARITY ME TRI C O N DATA S E T.
First image
Normally-exposed Normally-exposed Under-exposed
295
Second Image
Under-exposed Over-exposed Over-exposed
Corresponding errors (δx , δy )
B. Zitov, and J. Flusser, “Image registration methods: a survey,” Image and Vision Computing, vol. 21, pp. 977-1000, Oct. 2003. [2] L. G. Brown, “A survey of image registration techniques,” ACM Comput. Surv., vol. 24, pp. 325-376, Dec. 1992. [3] P.E. Debevec, and J. Malik, “Recovering High dynamic range radiance maps from photographs,” in proceeding of ACM SIGGRAPH 1997, pp. 369-378, 1997. [4] S. Mann, and R. Picard. “Being ’Undigital’ with digital camera: Extending dynamic range by combining differently exposed pictures,” in Proc. of IST’s 48th annual conference, pp. 442-448, May 1995. [5] T. Mintsunaga, and S. K. Nayer, “Radiometric self calibration,” in Proc. of Computer Vision and Pattern Recognition, vol. 1, pp. 374-380, 1999. [6] M. Robertson, S. Borman, and R. Stevenson, “Dynamic range improvement through multiple exposures,” in Proc. of the 1999 international conference on image processing (ICIP-99), vol. 3, pp. 159-163, Los Alamitos, CA, Oct. 24-28, 1999. [7] Greg Ward, “Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures,” journal of graphics tools, vol. 8(2), pp. 17-30, 2003. [8] E. Reinhard, G. Ward, S. Pattanaik, and P. Debvec, “High Dynamic Range Imaging Acquisition, Manupulation, and Display,” Morgan Kauf- mann, 2005. [9] A. Tomaszewska and R. Mantiuk, “Image Registration for Multiexposure High Dynamic Range Image Acquisition,” in Proc. of the International Conference on Computer Graphics, Visualization and Computer Vision, Plzen, Czech Republic, 2007. [10] Jones Loyd Ancile, and H. R. Condit, “Sunlight and skylight as determinants of Photographic exposure. I. Luminous density as determined by solar altitude and atmospheric conditions,” Journal of the Optical Society of America, vol. 38, no 2, pp. 123–178, Feb.
δx = 11, δy = 3 δx = 4, δy = 3 δx = −7, δy = 0
BUEST, Baddi
RIEECE -2015