Information about Super Resolution in Digital Image processing

Super Resolution in Digital Image processing

Super-resolution • convolutions, blur, and de-blurring • Bayesian methods • Wiener filtering and Markov Random Fields • sampling, aliasing, and interpolation • multiple (shifted) images • prior-based methods • MRFs • learned models • domain-specific models (faces)- Gary 3/7/2003 Super-Resolution 2

Linear systems Basic properties • homogeneity T[a X] T[X1+X2] • additivity = a T[X] = T[X1]+T[X2] • superposition T[aX1+bX2] = aT[X1]+bT[X2] Linear system ⇔ superposition Examples: • matrix operations (additions, multiplication) • convolutions 3/7/2003 Super-Resolution 3

Signals and linear operators Continuous Discrete Vector form I(x) I[k] or Ik I Discrete linear operator y=Ax Continuous linear operator: convolution integral g(x) = s h(ξ,x) f(ξ) dξ, h(ξ,x): impulse response g(x) = s h(ξ-x) f(ξ) dξ= [f * h](x) shift invariant 3/7/2003 Super-Resolution 4

2-D signals and convolutions Continuous Discrete I(x,y) I[k,l] or Ik,l 2-D convolutions (discrete) g[k,l] = ∑m,n f[m,n] h[k-m,l-n] = ∑m,n f[m,n] h1[k-m]h2[l-n] separable Gaussian kernel is separable and radial h(x,y) = (2πσ2)-1exp-(x2+y2)/σ2 3/7/2003 Super-Resolution 5

Convolution and blurring 3/7/2003 Super-Resolution 6

Separable binomial low-pass filter 3/7/2003 Super-Resolution 7

Fourier transforms Project onto a series of complex sinusoids F[m,n] = ∑k f[k,l] e-i 2π(km+ln) Properties: • shifting g(x-x0) ⇔ exp(-i 2πfxx0)G(fx) • differentiation dg(x)/dx ⇔ i 2πfxG(fx) • convolution 3/7/2003 [f * g](x) ⇔ [F G] (fx) Super-Resolution 8

Blurring examples Increasing amounts of blur + Fourier transform 3/7/2003 Super-Resolution 9

Sharpening Unsharp mask (darkroom photography): • remove some low-frequency content y’ = y + s (y – g * y) spatial (blur, sharp) 3/7/2003 Super-Resolution freq (blur,sharp) 10

Sharpening - result Unsharp mask: original, blur (σ=1), sharp(s=0, 1, 2) 3/7/2003 Super-Resolution 11

Deconvolution Filter by inverse of blur • easiest to do in the Fourier domain • problem: high-frequency noise amplification 3/7/2003 Super-Resolution 12

Bayesian modeling Use prior model for image and noise • y = g * x + n, x is original, y is blurred • p(x|y) = p(y|x)p(x) = exp(-|y – g*x|2/2σn-2) exp(-|x|2/2σx-2) • -log p(x|y) ∝ |y – g*x|2σn-2 + |x|2σx-2 where the norm || is summed squares over all pixels 3/7/2003 Super-Resolution 13

Parseval’s Theorem Energy equivalence in spatial ↔ frequency domain • |x|2 = |F(x)|2 • -log p(x|y) ∝ |Y(f) – G(f)X(f)|2σn-2 + |X(f)|2σx-2 • least squares solution (∂/∂X = 0) X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] 3/7/2003 Super-Resolution 14

Wiener filtering Optimal linear filter given noise and signal statistics • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • low frequencies: X(f) ≈ G-1(f)Y(f) boost by inverse gain (blur) X(f) ≈ G(f) σn-2σx2 Y(f) • high frequencies: attenuate by blur (gain) 3/7/2003 Super-Resolution 15

Wiener filtering – white noise prior Assume all frequencies equally likely • p(x) ~ N(0,σx2) • X(f) = G(f)Y(f) / [G2(f) + σn2/σx2] • solution is too noisy in high frequencies 3/7/2003 Super-Resolution 16

Wiener filtering – pink noise prior Assume frequency falloff (“natural statistics”) • p(X(f)) ~ N(0,|f|-βσx2) • X(f) = G(f)Y(f) / [G2(f) + |f|βσn2/σx2] • greater attenuation at high frequencies G(f) 3/7/2003 H(f) Super-Resolution 17

Markov Random Field modeling Use spatial neighborhood prior for image i • -log p(x) = ∑ij∈Cρ(xi-xj) where ρ(v) is a robust norm: • • • • j ρ(v) = v2: quadratic norm ↔ pink noise ρ(v) = |v|: total variation (popular with maths) ρ(v) = |v|β: natural statistics ρ(v) = v2,|v|: Huber norm [Schultz, R.R.; Stevenson, IEEE TIP, 1996] 3/7/2003 Super-Resolution 18

MRF estimation Set up discrete energy (quadratic or non-) • -log p(x|y) ∝ σn-2 |y – Gx|2 + ∑ij∈Cρ(xi-xj) where G is sparse convolution matrix • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) 3/7/2003 Super-Resolution 19

Sampling a signal • sampling: • creating a discrete signal from a continuous signal • downsampling (decimation) • subsampling a discrete signal • upsampling • introducing zeros between samples • aliasing • two sampled signals that differ in their original form (many → one mapping) 3/7/2003 Super-Resolution 20

Sampling interpolation 3/7/2003 Super-Resolution 21

Nyquist sampling theorem Signal to be (down-) sampled must have a bandwidth no larger than twice the sample frequency ωs = 2π / ns > 2 ω0 3/7/2003 Super-Resolution 22

Box filter (top hat) 3/7/2003 Super-Resolution 23

Ideal low-pass filter 3/7/2003 Super-Resolution 24

Simplified camera optics 1. 2. 3. 4. Blur = pill-box*Bessel2 (diffr.) ≈ Gaussian Integrate = box filter Sample = produce single digital sample Noise = additive white noise 3/7/2003 Super-Resolution 25

Aliasing Aliasing (“jaggies” and “crawl”) is present if blur amount < sampling (σ = 1) • shift each image in previous pipeline by 1 3/7/2003 Super-Resolution 26

Aliasing - less Less aliasing (“jaggies” and “crawl”) is present if blur amount ~ sampling (σ = 2) • shift each image in previous pipeline by 1 3/7/2003 Super-Resolution 27

Multi-image super-resolution Exploit aliasing to recover frequencies above Nyquist cutoff ∀ ∑kσn-2 |yk – Gkx|2 + ∑ij∈Cρ(xi-xj) where Gk are sparse convolution matrices • quadratic: solve sparse linear system • non-quadratic: use sparse non-linear least squares (Levenberg-Marquardt, gradient descent, conjugate gradient, …) • projection onto convex sets (POCS) 3/7/2003 Super-Resolution 28

Multi-image super-resolution Need: • accurate (sub-pixel) motion estimates (Wednesday’s lecture) • accurate models of blur (pre-filtering) • accurate photometry • no (or known) non-linear pre-processing (Bayer mosaics) • sufficient images and low-noise relative to amount of aliasing 3/7/2003 Super-Resolution 29

Prior-based Super-Resolution “Classical” non-Gaussian priors: • robust or natural statistics • maximum entropy (least blurry) • constant colors (black & white images) 3/7/2003 Super-Resolution 30

Example-based Super-Resolution William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, IEEE Computer Graphics and Applications, March/April, 2002 • learn the association between low-resolution patches and high-resolution patches • use Markov Network Model (another name for Markov Random Field) to encourage adjacent patch coherence 3/7/2003 Super-Resolution 31

Example-based Super-Resolution William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, IEEE Computer Graphics and Applications, March/April, 2002 3/7/2003 Super-Resolution 32

References – “classic” Irani, M. and Peleg. Improving Resolution by Image Registration. Graphical Models and Image Processing, 53(3), May 1991, 231-239. Schultz, R.R.; Stevenson, R.L. Extraction of high-resolution frames from video sequences. IEEE Trans. Image Proc., 5(6), Jun 1996, 996-1011. Elad, M.; Feuer, A.. Restoration of a single superresolution image from several blurred, noisy, and undersampled measured images. IEEE Trans. Image Proc., 6(12) , Dec 1997, 16461658. Elad, M.; Feuer, A.. Super-resolution reconstruction of image sequences. IEEE PAMI 21(9), Sep 1999, 817-834. Capel, D.; Zisserman, A.. Super-resolution enhancement of text image sequences. CVPR 2000, I-600-605 vol. 1. Chaudhuri, S. (editor). Super-Resolution Imaging. Kluwer Academic Publishers. 2001. 3/7/2003 Super-Resolution 33

References – strong priors Freeman, W.T.; Pasztor, E.C.. Learning low-level vision, CVPR 1999, 182-1189 vol.2 William T. Freeman, Thouis R. Jones, and Egon C. Pasztor, Example-based super-resolution, IEEE Computer Graphics and Applications, March/April, 2002 Baker, S.; Kanade, T. Hallucinating faces. Automatic Face Gesture Recognition, 2000, 83-88. Ce Liu; Heung-Yeung Shum; Chang-Shui Zhang. A two-step approach to hallucinating faces: global parametric model and local nonparametric model. CVPR 2001. I-192-8. 08/03/2014 Super-Resolution 34

... while in others—geometrical SR—the resolution of digital ... new Super lens. Geometrical or image-processing ... Image Super-Resolution ...

Read more

Image super -resolution ... is to accept the image degradations and use signal processing to post process ... the high resolution digital image (a) ...

Read more

Super-resolution (SR) is the process ... Superresolution of video acquired by a standard digital camera. Click on the images ... IEEE Transactions on Image ...

Read more

We conclude that super-resolution Raman ... The digital processing technology of the super-resolution ... on Image Processing ...

Read more

1 EE368: Digital Image Processing, Stanford University. ... Super-resolution image IV. DISCUSSION In this section, some of the artifacts and problems

Read more

New Applications of Super-resolution in ... 1.3.1 Super-Resolution in Low Radiation Digital X-ray ... The goal of super-resolution image processing is to ...

Read more

Superresolution FAQ. What is super-resolution. ... achieve with super-resolution processing, ... case of multi-image digital super-resolution: ...

Read more

Information about the open-access article 'Super-Resolution Raman Spectroscopy by Digital Image Processing' in DOAJ.

Read more

Edge processing by synthetic aperture superresolution in digital ... reasonable resolution limit. ... electronic image processing of the information ...

Read more

## Add a comment