Problem Definition
Hyperspectral imaging acquires images of the same scene at many different wavelengths at once, and it is widely used in agriculture, astronomy, surveillance, and mineralogy. Because of various hardware limitations, however, obtaining a high-resolution (HR) hyperspectral image remains a challenge. What follows shows how to reconstruct an HR hyperspectral image from a low-resolution (LR) hyperspectral image and an HR multispectral or RGB image of the same scene.
Given a pair of observed images, an LR hyperspectral image \(X \in \mathbb{R}^{L \times n}\) and an HR multispectral image \(Y \in \mathbb{R}^{3 \times N}\), the goal is to reconstruct the HR hyperspectral image \(Z \in \mathbb{R}^{L \times N}\), where \(N=W \times H, n=w \times h (w \ll W, h \ll H)\) are the numbers of pixels in the HR and LR images, respectively, and \(L \gg 3\) is the number of bands.
The two observed images can be written as linear transformations of the desired HR image:
\[X = ZH, \quad Y = PZ\]where \(H \in \mathbb{R}^{N \times n}\) is the degradation operator (blur and downsampling) associated with the LR image \(X\), and \(P\) is the transformation matrix that maps the HR hyperspectral image \(Z\) to its RGB representation \(Y\). Because there are far fewer equations than unknowns, estimating \(Z\) is an ill-posed inverse problem and must be regularized with prior knowledge.
Sparse-Representation Methods
Nonnegative Dictionary Learning
Using the sparse prior of a hyperspectral image, each pixel \(z_i \in \mathbb{R}^L\) can be written as a sparse linear combination of a dictionary \(D \in \mathbb{R}^{L \times K}\):
\[ z_i = D\alpha_i + e_i, \quad \text{s.t. } \|\alpha_i\|_0 < T \]where \(\alpha_i \in \mathbb{R}^K\) is the sparse coefficient vector and \(e_i\) is the approximation error. Each column of the dictionary \(D\) corresponds to the reflectance spectrum of a material in the scene. Under the linear mixing model, the LR hyperspectral image \(X\) can be written as
\[ x_i = D\beta_i + v_i, \quad \beta_i = \sum_{j \in W_i} h_j\alpha_j \]where \(W_i\) is a window centered at \(i\) and \(h_j\) is a weighting coefficient. At each location \(x_i\), only a few material constituents are present, so \(\beta_i\) is a sparse vector. To estimate the dictionary \(D\) and the sparse codes \(B=[\beta_1,...,\beta_n]\) at the same time, we minimize the following objective:
\[ (D, B) = \arg\min_{D,B} \frac{1}{2}\|X - DB\|_F^2 + \lambda\|B\|_1, \quad \text{s.t. } \beta_i \geq 0, d_k \geq 0 \]The objective is optimized by an efficient block-coordinate descent algorithm that updates one atom of the dictionary \(D\) at a time, which improves computational efficiency.
Clustering-Based Structured Sparse Coding
Once the dictionary \(D\) has been estimated, the sparse codes \(A=[\alpha_1,...,\alpha_N]\) of the target HR hyperspectral image \(Z\) can be estimated jointly from the HR multispectral image \(Y\) and the LR hyperspectral image \(X\):
\[ \begin{aligned} &Y = \tilde{D}A + W_1, \quad \tilde{D}=PD \\ &X = DAH + W_2 \end{aligned} \]We minimize the weighted reconstruction error together with an \(\ell_1\)-norm sparsity constraint:
\[ A = \arg\min_A \|Y - \tilde{D}A\|_F^2 + \|X - DAH\|_F^2 + \eta\|A\|_1, \quad \text{s.t. } \alpha_i \geq 0 \]To exploit spatial correlation among pixels in the hyperspectral image, we use a clustering-based nonnegative structured sparse representation (NSSR) model:
\[ \begin{aligned} A = \arg\min_A &\|Y - \tilde{D}A\|_F^2 + \|X - DAH\|_F^2\\ &+ \eta_1\sum_{q=1}^Q \sum_{i \in S_q} \|D\alpha_i - \mu_q\|_2^2 + \eta_2\|A\|_1,\quad \text{s.t. } \alpha_i \geq 0 \end{aligned} \]where \(\mu_q\) is the center of the \(q\)-th cluster and is updated recursively. \(S_q\) is the set of samples belonging to the \(q\)-th cluster, obtained by a \(k\)-nearest-neighbor search on the HR multispectral image. The model uses not only the sparse prior, but also a structural prior that reconstructed pixels should stay close to their cluster centers. The optimization problem above is solved efficiently by the alternating direction method of multipliers (ADMM).
In short, the method estimates the dictionary from the LR hyperspectral image, makes full use of the information in the HR multispectral image, and combines both sparse and structural priors.
