Non-local Neural Network
Please note that this post is for my probable research in the future to look back and review the materials on this topic without reading both papers[1, 2] twice completely.
The two convolutional and recurrent functions create blocks to process a local neighborhood at a time. This article will explain the non-local neural network which was used to denoising and then, by inspiring to be more economically and efficiently.
In 2005, in Europe, a Non-local means (NL-means) was proposed in order to measure the performance of digital image denoising methods.
Recovering the original image from a noisy one is the main aim of image denoising,
An appropriate easy manner to model the consequence of noise on a digital picture can be adding a gaussian white noise. Here are i.i.d. gaussian values with an average of 0 and variance of σ.
There are numerous methods and tools to discard the noise and recover the real picture. Despite this variety, it should be accentuated that a wide class shares the basic remark: Popular methods to accomplish the goal can be listed as below:
Averaging: This averaging can be carried out locally.
- such as the Gaussian smoothing model
- the anisotropic filtering
- the neighborhood filtering
The calculus of variations
- the Total Variation minimization
In the frequency domain:
- the empirical Wiener filters
- wavelet thresholding methods
NL-means algorithms
A discrete noisy image is given, the estimated value NL[v](i) for a pixel i can be calculated as a weighted average of all the pixels in the picture by the below equation:
Here, all weights are related to the similarity between pixels i and j and satisfy the common conditions that all weights should be 0 to 1, and their summation should be 1.
This parity illustrates the algorithm’s robustness since, in expectation, the Euclidean distance conserves the sequence of resemblance among pixels.
We can explain the weights as below:
Where the normalizing constant is:
h is the degree of filtering which governs the decay of the exponential function, and consequently the decay of the weights as a function of the Euclidean distances.
The NL-means, in addition to giving the comparison of the grey level in a single point, can compare the geometrical configuration in an entire neighborhood. This enables a more robust comparison than neighborhood filtering
A team in Carnegie Mellon University and Facebook AI Research, has been inspired by the above-mentioned classical non-local means and developed a non-local operation.
It is usual to utilize CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) architectures in long-range dependency modeling for sequential data (image, time-series, signal, …), and these both process a local neighborhood. Consequently, we have to apply these local operations frequently. This repeating has some limitations; being computationally inefficient and making optimization harder. We can consider non-local operations a productive, no complex, and generic component for long-range dependencies.
There are some benefits to using non-local neural networks:
- non-local operations instantly capture long-range dependencies by calculating interactions
- with lower complexity can accomplish best results
- non-local networks can blend with other architectures (convolutions, …) with no difficulty
the formula in non-local can be seen as below:
There is a distinct difference between the non-local and fully connected (fc) layer. In the non-local, it is about the relationships between different locations, while, in the fully connected layer, learned weights are used.
The non-local block can be expressed as below:
We can consider non-local neural networks as a new class of neural networks that take long-range dependencies by some calculations. This type of neural network can be mixed with other types of architectures. These non-local networks are appropriate for sequential data such as video, real-time, image, audio, etc. there will be a great potential for non-local networks to play a significant role in the future.
If any errors are found, please email me at rezayazdanfar1111@gmail.com. Meanwhile, follow me on my Twitter here, and visit my LinkedIn here. In the end, if you found it useful and want to pursue future articles please follow me in the medium.
1. Buades, A., B. Coll, and J.-M. Morel. A non-local algorithm for image denoising. in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05). 2005. IEEE.
2. Wang, X., et al. Non-local neural networks. in Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.
