Image Inpainting

Image Inpainting Işık Barış Fidaner

Image Inpainting • Filling unknown area on the image by using surrounding information.

?

Unknown area

?

Known area

?

Image Inpainting • Filling unknown area on the image by using surrounding information.

Inpainting direction

Known area

Unknown area

Three related fields of study Texture Synthesis Variational Image Inpainting

Constrained Texture Synthesis

Image Completion

Variational Image Inpainting • Filling unknown area on the image by using surrounding structure information.

Structure inpainting Unkown area Known area

Variational Image Inpainting Original image Inpainted image

Example:

Structure inpainting Unkown area Known area

Texture Synthesis • An input sample of a texture is given, and the goal is to produce more of that texture

Texture synthesis Texture sample

More of that texture

Texture Synthesis Input sample

Synthesis output

Example:

Texture synthesis Texture sample

More of that texture

Constrained Texture Synthesis • Filling unknown area on the image by using surrounding texture information.

Constrained synthesis Unkown area Known area

Constrained Texture Synthesis Input image

Inpainted image

Example: (illustrative)

Constrained synthesis Unkown area Known area

Image Completion • Filling unknown area on the image by using both texture and structure information.

Texture and structure inpainting Unkown area Known area

Image Completion Input image

Completed image

Example:

Texture and structure inpainting Unkown area Known area

Applications • Variational inpainting: – Scaling-up images – Reconstructing old photographs – Removal of overlaid text or graphics etc.

• Texture Synthesis: – 3D surface covering – Hole filling

• Image Completion – Removing unwanted objects from images – Filling-in the image blocks that are lost in transmission

Image Inpainting requires…

• Good continuation of image geometry into and out of the area • Believable reproduction of textures • Good continuation of the structure of composite textures

Image Inpainting approaches • There are two main approaches: – Geometric image inpainting methods are based on continuation of geometric structures.

– In contrast, texture synthesis methods fill image by reproducing textures.

– There are also combined methods

Original image Inpainted image

Geometric image inpainting • First determine geometric structure defined by edge lines or isophotes on the image.

• Model the geometry (usually with a partial differential equation).

• Fill the spaces by propogating known information inward by the equation.

Geometric image inpainting • “Image Inpainting” Bertalmio et al. 2000 • Geometric information is given by L (Laplacian of image).

• N is normal to unknown area boundary.

• Change of L in direction of N is painted every iteration, propogating geometric information into the unknown area, until convergence occurs.

Geometric image inpainting

Texture Synthesis • A texture sample is given or extracted from known image areas.

• We must produce new texture that visually appears similar to the sample • For an unknown region part, find a similar part in the texture sample.

• Copy a pixel or a block from the sample into the unknown area.

Texture Synthesis • “Texture Synthesis by Non-parametric Sampling” Efros and Leung 1999 • Texture is modelled as a Markov Random Field • For each unknown pixel, a corresponding pixel in the sample texture is found, such that surrounding rectangular patches are similar (Patch comparison is weighted.

Center part of the patch has more effect).

Texture Synthesis

Simultaneous texture & geometry • “Simultaneous Structure and Texture Image Inpainting” Bertalmio et al. 2003 • decompose the image into the sum of two functions with different basic characteristics, • reconstruct each one of these functions separately with structure and texture filling-in algorithms.

Simultaneous texture & geometry

Exemplar-based inpainting • “Region Filling and Object Removal by Exemplar-Based Image Inpainting” Criminisi et al. 2004 • Fill the area by copying similar patches from other parts of the image (believable repetition of texture) • Parts with more geometric information is filled first (good continuation of geometric structure)

Exemplar-based inpainting

Exemplar-based inpainting

Simple application

Simple application

Simple application 1 • Fill unknown area from a sample texture • Compare each unknown pixel to 50 random 5x5 patches from sample texture • Copy pixel by pixel

Simple application 1

Simple application 2 • Based on “Synthesizing Natural Textures” Ashikhmin 2001 • For each unknown pixel (row by row), compare to 4 candidate patches (3x3) from sample texture, fill with best match • Candidate patches are “shifted” from neighbour pixels

Simple application 2

Simple application 3 • Based on exemplar-based inpainting • Fill unknown area from a sample texture • Pick a 9x9 patch on boundary of the unknown area • Compare it to 500 random patches from texture sample • Copy best fitting patch

Simple application 3