An image denoising is an algorithm that learns what is noise (in some noisy image) and how to remove it, based into the true signal / original (image without noisy). The results are images very close to the true ones, for example, as in the image below:
Autoencoders are based on Neural Networks (NNs) and are known as Convolutional Neural Networks (CNNs or convnets). A convnet is a Deep Learning algorithm which takes an input image, assign importance (learnable weight, biases and retains spatial relationships in the data into each one of theirs layers) to various aspects/parts in the image and is able to differentiate/reconstruct the same.
The general idea behind this kind of code can be visualized here:
Then, the autoencoder compreehends an encoder and a decoder. The encoder does the encoding process, i.e., transforms the image into a compressed representation at the same time that starts the noisy reduction. Then, the compressed representation goes to decoder that performs the decoder process, restoring the image to its true and recognizable shape. At the end of the process, we remove almost all noise in the image.
import keras
from keras import layers
from keras.datasets import mnist
import numpy as np
import matplotlib.pyplot as plt
This dataset is mostly used in classification problems, that is why they have the images and a second information, the labels of each image.
But, as we are going to map digits images to clean them, we are not going to use the labels.
In this way, our process is going to take the noised images and clear the same based into the true images as targets.
(x_train, _), (x_test, _) = mnist.load_data()
x_train.shape, x_test.shape
((60000, 28, 28), (10000, 28, 28))
dimension = x_train.shape[1]
Because here we are doing a simple example, we will use a fraction of the complete dataset: just 1000 images, dividing if on 700 as train and 300 images as test.
num_data = 1000
frac_train = 0.7
frac_test = 0.3
x_train = x_train[0:int(frac_train*num_data)]
x_test = x_test[0:int(frac_test*num_data)]
We are going to normalize the images between 0 and 1 and to reshape them.
norm_factor = 255.
x_train = x_train.astype('float32')/norm_factor
x_test = x_test.astype('float32')/norm_factor
x_train = np.reshape(x_train, (len(x_train), dimension, dimension, 1))
x_test = np.reshape(x_test, (len(x_test), dimension, dimension, 1))
Here, we need to noisy the images, then, we apply a Gaussian noisy matrix and clip the images between 0 and 1.
noise_factor = 0.5
x_train_noisy = x_train + noise_factor * np.random.normal(loc = 0.0, scale = 1.0, size = x_train.shape)
x_test_noisy = x_test + noise_factor * np.random.normal(loc = 0.0, scale = 1.0, size = x_test.shape)
x_train_noisy = np.clip(x_train_noisy, 0., 1.)
x_test_noisy = np.clip(x_test_noisy, 0., 1.)
Visualizing some noise images images.
n = 3
for i in range(n):
fig, axes = plt.subplots(1, 2)
fig.set_size_inches(5, 5)
axes[0].set_title('True image')
im0 = axes[0].imshow(x_test[i].reshape(dimension, dimension), cmap = 'Reds')
axes[1].set_title('Noisy image')
im1 = axes[1].imshow(x_test_noisy[i].reshape(dimension, dimension), cmap = 'Reds')
Defining the input images for the autoencoder.
input_img = keras.Input(shape = (dimension, dimension, 1))
x = layers.Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu', padding = 'same')(input_img)
x = layers.MaxPooling2D(pool_size = (2, 2), padding = 'same')(x)
x = layers.Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu', padding = 'same')(x)
encoded = layers.MaxPooling2D(pool_size = (2, 2), padding = 'same')(x)
x = layers.Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu', padding = 'same')(encoded)
x = layers.UpSampling2D(size = (2, 2))(x)
x = layers.Conv2D(filters = 32, kernel_size = (3, 3), activation = 'relu', padding = 'same')(x)
x = layers.UpSampling2D(size = (2, 2))(x)
decoded = layers.Conv2D(filters = 1, kernel_size = (3, 3), activation = 'sigmoid', padding = 'same')(x)
autoencoder = keras.Model(input_img, decoded)
Visualizing the autoencoder.
autoencoder.summary()
Model: "functional_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) [(None, 28, 28, 1)] 0 _________________________________________________________________ conv2d (Conv2D) (None, 28, 28, 32) 320 _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 14, 14, 32) 0 _________________________________________________________________ conv2d_1 (Conv2D) (None, 14, 14, 32) 9248 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 7, 7, 32) 0 _________________________________________________________________ conv2d_2 (Conv2D) (None, 7, 7, 32) 9248 _________________________________________________________________ up_sampling2d (UpSampling2D) (None, 14, 14, 32) 0 _________________________________________________________________ conv2d_3 (Conv2D) (None, 14, 14, 32) 9248 _________________________________________________________________ up_sampling2d_1 (UpSampling2 (None, 28, 28, 32) 0 _________________________________________________________________ conv2d_4 (Conv2D) (None, 28, 28, 1) 289 ================================================================= Total params: 28,353 Trainable params: 28,353 Non-trainable params: 0 _________________________________________________________________
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
validation_split = 0.8
history = autoencoder.fit(x_train_noisy, x_train, epochs = 40, batch_size = 20, shuffle = True, validation_split = validation_split)
Epoch 1/40 7/7 [==============================] - 1s 72ms/step - loss: 0.5926 - val_loss: 0.4993 Epoch 2/40 7/7 [==============================] - 0s 57ms/step - loss: 0.4989 - val_loss: 0.4638 Epoch 3/40 7/7 [==============================] - 0s 54ms/step - loss: 0.4516 - val_loss: 0.4265 Epoch 4/40 7/7 [==============================] - 0s 53ms/step - loss: 0.4046 - val_loss: 0.3686 Epoch 5/40 7/7 [==============================] - 0s 53ms/step - loss: 0.3384 - val_loss: 0.2967 Epoch 6/40 7/7 [==============================] - 0s 53ms/step - loss: 0.2699 - val_loss: 0.2410 Epoch 7/40 7/7 [==============================] - 0s 53ms/step - loss: 0.2273 - val_loss: 0.2148 Epoch 8/40 7/7 [==============================] - 0s 53ms/step - loss: 0.2075 - val_loss: 0.2056 Epoch 9/40 7/7 [==============================] - 0s 55ms/step - loss: 0.1972 - val_loss: 0.1954 Epoch 10/40 7/7 [==============================] - 0s 57ms/step - loss: 0.1873 - val_loss: 0.1881 Epoch 11/40 7/7 [==============================] - 0s 57ms/step - loss: 0.1797 - val_loss: 0.1811 Epoch 12/40 7/7 [==============================] - 0s 71ms/step - loss: 0.1737 - val_loss: 0.1757 Epoch 13/40 7/7 [==============================] - 0s 68ms/step - loss: 0.1693 - val_loss: 0.1724 Epoch 14/40 7/7 [==============================] - 0s 56ms/step - loss: 0.1649 - val_loss: 0.1678 Epoch 15/40 7/7 [==============================] - 1s 76ms/step - loss: 0.1609 - val_loss: 0.1641 Epoch 16/40 7/7 [==============================] - 1s 77ms/step - loss: 0.1581 - val_loss: 0.1623 Epoch 17/40 7/7 [==============================] - 0s 70ms/step - loss: 0.1548 - val_loss: 0.1578 Epoch 18/40 7/7 [==============================] - 0s 53ms/step - loss: 0.1521 - val_loss: 0.1570 Epoch 19/40 7/7 [==============================] - 1s 73ms/step - loss: 0.1502 - val_loss: 0.1548 Epoch 20/40 7/7 [==============================] - 1s 72ms/step - loss: 0.1478 - val_loss: 0.1514 Epoch 21/40 7/7 [==============================] - 0s 58ms/step - loss: 0.1454 - val_loss: 0.1489 Epoch 22/40 7/7 [==============================] - 0s 59ms/step - loss: 0.1424 - val_loss: 0.1472 Epoch 23/40 7/7 [==============================] - 0s 68ms/step - loss: 0.1403 - val_loss: 0.1447 Epoch 24/40 7/7 [==============================] - 1s 72ms/step - loss: 0.1379 - val_loss: 0.1442 Epoch 25/40 7/7 [==============================] - 0s 68ms/step - loss: 0.1363 - val_loss: 0.1414 Epoch 26/40 7/7 [==============================] - 0s 62ms/step - loss: 0.1344 - val_loss: 0.1404 Epoch 27/40 7/7 [==============================] - 0s 55ms/step - loss: 0.1325 - val_loss: 0.1387 Epoch 28/40 7/7 [==============================] - 0s 56ms/step - loss: 0.1309 - val_loss: 0.1376 Epoch 29/40 7/7 [==============================] - 0s 58ms/step - loss: 0.1293 - val_loss: 0.1363 Epoch 30/40 7/7 [==============================] - 0s 57ms/step - loss: 0.1278 - val_loss: 0.1355 Epoch 31/40 7/7 [==============================] - 1s 73ms/step - loss: 0.1265 - val_loss: 0.1343 Epoch 32/40 7/7 [==============================] - 0s 55ms/step - loss: 0.1254 - val_loss: 0.1340 Epoch 33/40 7/7 [==============================] - 0s 56ms/step - loss: 0.1241 - val_loss: 0.1338 Epoch 34/40 7/7 [==============================] - 0s 71ms/step - loss: 0.1245 - val_loss: 0.1353 Epoch 35/40 7/7 [==============================] - 0s 60ms/step - loss: 0.1242 - val_loss: 0.1357 Epoch 36/40 7/7 [==============================] - 0s 65ms/step - loss: 0.1246 - val_loss: 0.1391 Epoch 37/40 7/7 [==============================] - 0s 55ms/step - loss: 0.1249 - val_loss: 0.1370 Epoch 38/40 7/7 [==============================] - 0s 66ms/step - loss: 0.1233 - val_loss: 0.1355 Epoch 39/40 7/7 [==============================] - 0s 53ms/step - loss: 0.1237 - val_loss: 0.1298 Epoch 40/40 7/7 [==============================] - 0s 59ms/step - loss: 0.1201 - val_loss: 0.1319
history.history.keys()
dict_keys(['loss', 'val_loss'])
train_loss = history.history['loss']
train_val_loss = history.history['val_loss']
epochs = range(1, len(train_loss) + 1)
Visualizing the history of the training.
plt.figure(dpi = 100)
plt.plot(epochs, train_loss, label = 'Loss')
plt.plot(epochs, train_val_loss, 'o', label = 'Val loss')
plt.title('Training and validation metrics')
plt.legend()
plt.savefig('history.png')
all_denoised_images = autoencoder.predict(x_test_noisy)
test_loss = autoencoder.evaluate(x_test_noisy, x_test, batch_size = 20)
test_loss
15/15 [==============================] - 0s 8ms/step - loss: 0.1299
0.12987352907657623
Here, we can compare our visual results looking side by side the noisy, targets and denoised images.
n = 3
for i in range(n):
fig, axes = plt.subplots(1, 3)
fig.set_size_inches(8, 2)
axes[0].set_title('Noisy image')
im0 = axes[0].imshow(x_test_noisy[i].reshape(dimension, dimension), cmap = 'Reds')
axes[1].set_title('Target image')
im1 = axes[1].imshow(x_test[i].reshape(dimension, dimension), cmap = 'Reds')
axes[2].set_title('Denoised image')
im2 = axes[2].imshow(all_denoised_images[i].reshape(dimension, dimension), cmap = 'Reds')
plt.savefig(f'comparison-{i}.png')
