Project IA Deep Learning - Supervised

DataSet MNIST Fashion.

Philippe Jadoul - january 2020

Fashion MNIST - Network Dense

Imports

import sys
import os
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf

from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import load_model
from sklearn.model_selection import train_test_split

Import my_fonctions

sys.path.append("/home/ia-serveur/ia/my_lib_functions")
#sys.path.append("/home/ia/ia/my_lib_functions")
from my_functions import *

sys.path.append('./my_plot_results')
from my_functions_plot import *

Tests All

assert hasattr(tf, "function") # Be sure to use tensorflow 2.X
print('tensorflow :',tf.__version__)
first_function()

tensorflow : 2.2.0
This is the first function

Load the dataset: Fashion MNIST

dataset which contains 70,000 grayscale images in 10 categories. The images show individual articles of clothing at low resolution (28 by 28 pixels), train_images = 60.000, train_test = 10000

Import and load the Fashion MNIST data from local folder data - (Test) files compressed *.gz

• function - np_load_mnist(path, header='train' or 't10k') - My fubctions Library
• Not Actvate (Only my test local

Import and load the Fashion MNIST data directly from TensorFlow:

fashion_mnist = keras.datasets.fashion_mnist
(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()

print('Shape train_images & train_labels',train_images.shape, train_labels.shape )
print('Shape test_images & Test_images  ',test_images.shape, test_labels.shape )

Shape train_images & train_labels (60000, 28, 28) (60000,)
Shape test_images & Test_images   (10000, 28, 28) (10000,)

max = 0
if max > 0 :
    train_images = train_images[:max]
    train_labels = train_labels[:max]
else:
    train_images = train_images
    train_labels = train_labels

print('Shape train_images & train_labels',train_images.shape, train_labels.shape )
print('Shape test_images & Test_images  ',test_images.shape, test_labels.shape )

Shape train_images & train_labels (60000, 28, 28) (60000,)
Shape test_images & Test_images   (10000, 28, 28) (10000,)

Create Validation Dataset

train_images, train_images_validate, train_labels, train_labels_validate = train_test_split(
                                                    train_images, train_labels, test_size=0.2, random_state=1)

print('Shape train_images & train_labels                    '
          ,train_images.shape, train_labels.shape )
print('Shape train_images_validate & train_labels_validate  '
          ,train_images_validate.shape, train_labels_validate.shape )

Shape train_images & train_labels                     (48000, 28, 28) (48000,)
Shape train_images_validate & train_labels_validate   (12000, 28, 28) (12000,)

print('train_images :',train_images.shape)
print('train_labels :',train_labels.shape)
print('train_images_validate :',train_images_validate.shape)
print('train_labels_validate :',train_labels_validate.shape)
print('test_images :',test_images.shape)
print('test_labels :',test_labels.shape)
print(train_images.dtype)
print(train_labels.dtype)

train_images : (48000, 28, 28)
train_labels : (48000,)
train_images_validate : (12000, 28, 28)
train_labels_validate : (12000,)
test_images : (10000, 28, 28)
test_labels : (10000,)
uint8
uint8

plt.figure(figsize=(12, 4))
plt.subplot(1, 3, 1)
plt.imshow(train_images[0])
plt.colorbar()
plt.grid(False)
plt.subplot(1, 3, 2)
plt.imshow(train_images[0], cmap=plt.cm.binary)
plt.colorbar()
plt.grid(False)
plt.subplot(1, 3, 3)
plt.imshow(train_images[0], cmap=plt.cm.gray)
plt.colorbar()
plt.grid(False)
plt.show()

Distribution Data Before Normalization

max_h = np.max(train_images[0])
min_h = np.min(train_images[0]) 
print('max = ',max_h)
print('min = ',min_h)
plt.hist(train_images[0].ravel(), color='gray', bins=100)
plt.title('Histogram Distribution Data')
plt.xlabel('Valeur Data')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

max =  255
min =  0

Flatten Data images - (28x28) - > (784) Pixels

# Reshape the dataset and convert to float
X_train = train_images.reshape(-1, 784)
X_train = X_train.astype(float) 
X_test = test_images.reshape(-1, 784)
X_test = X_test.astype(float) 

X_train_validate = train_images_validate.reshape(-1, 784)
X_train_validate = X_train_validate.astype(float) 

print('X_train.shape               :',X_train.shape)
print('X_train_validate.shape      :',X_train_validate.shape)
print('X_test.shape                :',X_test.shape)
print('train_images.shape          :',train_images.shape)
print('train_images_validate.shape :',train_images_validate.shape)

X_train.shape               : (48000, 784)
X_train_validate.shape      : (12000, 784)
X_test.shape                : (10000, 784)
train_images.shape          : (48000, 28, 28)
train_images_validate.shape : (12000, 28, 28)

Normalization - Simple

train_images = train_images / 255.0
test_images = test_images / 255.0

plt.figure(figsize=(12, 4))
plt.subplot(1, 3, 1)
plt.imshow(train_images[0])
plt.colorbar()
plt.grid(False)
plt.subplot(1, 3, 2)
plt.imshow(train_images[0], cmap=plt.cm.binary)
plt.colorbar()
plt.grid(False)
plt.subplot(1, 3, 3)
plt.imshow(train_images[0], cmap=plt.cm.gray)
plt.colorbar()
plt.grid(False)
plt.show()

Distribution Data After Simple Normalization

max_h = np.max(train_images[0])
min_h = np.min(train_images[0]) 
print('max = ',max_h)
print('min = ',min_h)
plt.hist(train_images[0].ravel(), color='gray', bins=100)
plt.title('Histogram Distribution Data')
plt.xlabel('Valeur Data')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

max =  1.0
min =  0.0

Type of class_names :

Label	Class
0	T-shirt/top
1	Trouser
2	Pullover
3	Dress
4	Coat
5	Sandal
6	Shirt
7	Sneaker
8	Bag
9	Ankle boot

Normalization - StandardScater - $z = \frac{(X - \overline{U})}{\sigma}$

• Le score standard d'un echantillon X est calcule comme suit :
  • U est la moyenne des echantillons d'apprentisage.
  • $\sigma$ est l'ecart type des echantillons d'apprentisage

from sklearn.preprocessing import StandardScaler
print("Before Mean and std of images :", X_train.mean(), ' - ', X_train.std())
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)
X_train_validate = scaler.transform(X_train_validate)
print("After  Mean and std of images :", X_train.mean(), ' - ', X_train.std())

Before Mean and std of images : 72.86190226403062  -  89.9435889780605
After  Mean and std of images : -2.235551124415281e-18  -  1.000000000000003

plt.figure(figsize=(12, 4))
plt.subplot(1, 3, 1)
plt.imshow(np.reshape(X_train[0], (28, 28)))
plt.colorbar()
plt.grid(False)
plt.subplot(1, 3, 2)
plt.imshow(np.reshape(X_train[0], (28, 28)), cmap=plt.cm.binary)
plt.colorbar()
plt.grid(False)
plt.subplot(1, 3, 3)
plt.imshow(np.reshape(X_train[0], (28, 28)), cmap=plt.cm.gray)
plt.colorbar()
plt.grid(False)
plt.hist(train_images[0])
plt.show()

Distribution Data Aflter StandardScater Normalization

max_h = np.max(X_train[0])
min_h = np.min(X_train[0]) 
print('max = ',max_h)
print('min = ',min_h)
print("Mean and std of data train", "- Mean= ",X_train.mean()," - std= ", X_train.std())
plt.hist(X_train[0].ravel(), color='gray', bins=100)
plt.title('Histogram Distribution Data')
plt.xlabel('Valeur Data')
plt.ylabel('Frequency')
plt.grid(True)
plt.show()

max =  6.113497281677134
min =  -1.8843069488530648
Mean and std of data train - Mean=  -2.235551124415281e-18  - std=  1.000000000000003

One-hot encoding

unique, counts = np.unique(train_labels, return_counts=True)
n_classes = len(unique)

Y_train = tf.keras.utils.to_categorical(train_labels, n_classes )
Y_test = tf.keras.utils.to_categorical(test_labels, n_classes )
Y_train_validate = tf.keras.utils.to_categorical(train_labels_validate, n_classes )


print('Diff Classes :',unique)
print('nb Classes :',n_classes)
print('Shape before one-hot encoding :', train_labels.shape)
print('Labels 0 :',train_labels[0])
print('Shape after one-hot encoding :', Y_train.shape)
print('Labels [0] Y_Train :',Y_train[0])
print('Labels [0] Y_Test  :',Y_test[0])
print('Labels [0] Y_Train_validate :',Y_train_validate[0])

Diff Classes : [0 1 2 3 4 5 6 7 8 9]
nb Classes : 10
Shape before one-hot encoding : (48000,)
Labels 0 : 9
Shape after one-hot encoding : (48000, 10)
Labels [0] Y_Train : [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
Labels [0] Y_Test  : [0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]
Labels [0] Y_Train_validate : [0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]

List targets_names - Labels og Classes

targets_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

To verify that the data is in the correct format and that you're ready to build and train the network, let's display the first 25 images from the training set and display the class name below each image.

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(train_images[i], cmap=plt.cm.binary)
    plt.xlabel(targets_names[train_labels[i]])
plt.show()

Create the model

Create the Dense model

# Create the model
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(256, activation="relu", dtype=tf.float64))
model.add(tf.keras.layers.Dense(128, activation="relu", dtype=tf.float64))
model.add(tf.keras.layers.Dense(10, activation="softmax", dtype=tf.float64))

Compiling the sequentail model - Dense

• Loss function —This measures how accurate the model is during training with (spare_categorical_crossentropy)
  • (categorical) - possibilité d'avoir plus de deux classe.
  • (spare) - un seul entier par classe.
• opimizer function —This is how the model is updated based on the data it sees and its loss function.
  • (sgd) - minimize the function with mrthod Stchastic Gradien Descent.

• Metrics — Used to monitor the training and testing steps.

  • (accuracy) - Precision (also called predictve value)

• Si les classes ou Labels sont code sur un digit (ex: 9 -> [9] alors

  • loss='sparse_categorical_crossentropy'
• Si les classes ou Labels sont code en One-hot encoding (ex: 9 -> [0,0,0,0,0,0,0,0,1] alors
  • loss='"categorical_crossentropy'

Function Compile Model

# Compile the model
def compile_model():
    model.compile(
        loss="categorical_crossentropy",
        optimizer="sgd",
        metrics=["accuracy"]
    )

Compile Model

compile_model()

Train the model

• epochs : number pass on training dataset.
• batchs_size : divides dataset into x packages.
• validation_split : create a validation dataset x% of the training dataset.

(1) Training the model - History model fit

history = model.fit(X_train, Y_train, epochs=50,batch_size=None, validation_split=0.0, verbose=False )

history = model.fit(X_train, Y_train, epochs=50,batch_size=None, validation_split=0.0, verbose=False )

Graph - fig-1.png

plotting_train_metrics(history, save=False, name='fig-1.png')

Train max accuracy = 99.59 %
Train max loss = 2.10 %

Model Summary

model.summary()

Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                multiple                  200960    
_________________________________________________________________
dense_1 (Dense)              multiple                  32896     
_________________________________________________________________
dense_2 (Dense)              multiple                  1290      
=================================================================
Total params: 235,146
Trainable params: 235,146
Non-trainable params: 0
_________________________________________________________________

Function Reset Model Dense (Re-Create & Re-Compile)

def reset_model():
    # re-Create the model
    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.Dense(256, activation="relu", dtype=tf.float64))
    model.add(tf.keras.layers.Dense(128, activation="relu", dtype=tf.float64))
    model.add(tf.keras.layers.Dense(10, activation="softmax", dtype=tf.float64))
    
    return model

Reset Model Dense

del model
model=reset_model()
compile_model()

Evaluate the Models's Performance & optimalisation metrics

(2) Training with split Data Train 20% - epochs 50 - history model fit

history = model.fit(X_train, Y_train, epochs=50,batch_size=None,validation_split=0.2, verbose=False)

history = model.fit(X_train, Y_train, epochs=50,batch_size=None,validation_split=0.2, verbose=False)

Graph - fig-2.png

plotting_train_metrics(history, save=False, name='fig-2.png')

Train max accuracy = 99.69 %  <--> Train max val_accuracy = 88.76 %
Train max loss     = 1.87 %  <--> Train max val_loss     = 51.36 %

Reset Model Dense

del model
model=reset_model()
compile_model()

(3) Training the model - Amelioration witn batch_size 256 - epochs 50 - validation_split 20%

history = model.fit(X_train, Y_train, epochs=50,batch_size=256, validation_split=0.2, verbose=False)

history = model.fit(X_train, Y_train, epochs=50,batch_size=256, validation_split=0.2, verbose=False)

Graph - fig-3.png

plotting_train_metrics(history, save=False, name='fig-3.png')

Train max accuracy = 91.91 %  <--> Train max val_accuracy = 87.98 %
Train max loss     = 23.31 %  <--> Train max val_loss     = 33.91 %

Reset Model Dense

del model
model=reset_model()
compile_model()

(4) Training with separate Validation Data & epochs 50

history = model.fit(X_train, Y_train, epochs=50,batch_size=None, validation_data=(X_train_validate,Y_train_validate), verbose=False )

history = model.fit(X_train, Y_train, epochs=50,batch_size=None, 
                  validation_data=(X_train_validate,Y_train_validate), verbose=False )

Graph - fig-4.png

plotting_train_metrics(history, save=False, name='fig-4.png')

Train max accuracy = 99.70 %  <--> Train max val_accuracy = 89.11 %
Train max loss     = 1.90 %  <--> Train max val_loss     = 51.17 %

Reset Model Dense

del model
model=reset_model()
compile_model()

(5) Training with Validation Data Set - epochs 50 batch size 256

history = model.fit(X_train, Y_train, epochs=50,batch_size=256, validation_data=(X_train_validate,Y_train_validate), verbose=True )

history = model.fit(X_train, Y_train, epochs=50,batch_size=256, 
                 validation_data=(X_train_validate,Y_train_validate), verbose=False )

Graph - fig-5.png

plotting_train_metrics(history, save=False, name='fig-5.png')

Train max accuracy = 92.24 %  <--> Train max val_accuracy = 88.26 %
Train max loss     = 22.32 %  <--> Train max val_loss     = 32.53 %

Reset Model Dense

del model
model=reset_model()
compile_model()

(6) Training with Validation Data Set - epochs 256 - batch_size 512

history = model.fit(X_train, Y_train, epochs=100,batch_size=256, validation_data=(X_train_validate,Y_train_validate), verbose=False )

history = model.fit(X_train, Y_train, epochs=256,batch_size=512, 
                   validation_data=(X_train_validate,Y_train_validate), verbose=False )

Graph - fig-6.png

plotting_train_metrics(history, save=False, name='fig-6.png')

Train max accuracy = 97.17 %  <--> Train max val_accuracy = 89.17 %
Train max loss     = 10.15 %  <--> Train max val_loss     = 35.06 %

function plotting the metrics of train dataset accuracy & loss - from my library

• function -plotting_train_metrics(history, save=False, name='fig-.png')

The Best Model Seems to be the (5) - Re-Training - Metrics (5)

Reset Model Dense

del model
model=reset_model()
compile_model()

(5) Re-Training with Validation Data Set - epochs 50 batch size 256

history = model.fit(X_train, Y_train, epochs=50,batch_size=256, validation_data=(X_train_validate,Y_train_validate), verbose=True )

history = model.fit(X_train, Y_train, epochs=50,batch_size=256, 
                 validation_data=(X_train_validate,Y_train_validate), verbose=False )

history_dict = history.history
print(history_dict.keys())

dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

Best Model (5)

plotting_train_metrics(history, save=False, name='best-5.png')

Train max accuracy = 92.12 %  <--> Train max val_accuracy = 88.22 %
Train max loss     = 22.81 %  <--> Train max val_loss     = 32.54 %

Model Summary

model.summary()

Model: "sequential_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_18 (Dense)             multiple                  200960    
_________________________________________________________________
dense_19 (Dense)             multiple                  32896     
_________________________________________________________________
dense_20 (Dense)             multiple                  1290      
=================================================================
Total params: 235,146
Trainable params: 235,146
Non-trainable params: 0
_________________________________________________________________

Evaluate the Models' Performance with the best current metrics

compare how the model performs on the test dataset:

results = model.evaluate(X_test, Y_test, verbose=True)
print('test_loss, test_acc', results)
print('test_accuracy = {t1:3.2f} %'.format(t1=results[1]*100.0))
print('test_loss     = {t1:3.2f} %'.format(t1=results[0]*100.0))

313/313 [==============================] - 1s 2ms/step - loss: 0.3567 - accuracy: 0.8749
test_loss, test_acc [0.35673987865448, 0.8748999834060669]
test_accuracy = 87.49 %
test_loss     = 35.67 %

save the model after training (5)

# saving the model
path = './results'
model_name = 'mnist_fashion.h5'
model_path = os.path.join(path, model_name)
model.save(model_path)
print('Saved trained model at %s ' % model_path)

Saved trained model at ./results/mnist_fashion.h5 

Delete the model

del model

Load the Model (5)

model = tf.keras.models.load_model(model_path)

Model Summary

model.summary()

Model: "sequential_6"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_18 (Dense)             multiple                  200960    
_________________________________________________________________
dense_19 (Dense)             multiple                  32896     
_________________________________________________________________
dense_20 (Dense)             multiple                  1290      
=================================================================
Total params: 235,146
Trainable params: 235,146
Non-trainable params: 0
_________________________________________________________________

Re-Evaluate the Models' Performance with the best current metrics (5)

compare how the model performs on the test dataset:

results = model.evaluate(X_test, Y_test, verbose=True)
print('test_loss, test_acc', results)
print('test_accuracy = {t1:3.2f} %'.format(t1=results[1]*100.0))
print('test_loss     = {t1:3.2f} %'.format(t1=results[0]*100.0))

313/313 [==============================] - 1s 2ms/step - loss: 0.3567 - accuracy: 0.8749
test_loss, test_acc [0.35673987865448, 0.8748999834060669]
test_accuracy = 87.49 %
test_loss     = 35.67 %

Make predictions

With the model trained, you can use it to make predictions about some images.

predictions = model.predict(X_test, batch_size=None, verbose=False)
print('Shape Prediction :',predictions.shape)

Shape Prediction : (10000, 10)

resulat prediction image index 12 - (probability - class ( 0 to 9 )

predictions[12]

array([9.79079234e-06, 1.35234700e-04, 5.43827248e-05, 4.81204603e-02,
       1.60205146e-04, 2.44440821e-01, 2.32082001e-05, 6.83973378e-01,
       2.25804018e-02, 5.02117539e-04])

position (index) of the maximum probabability

print('Argmax :',np.argmax(predictions[12]))

Argmax : 7

one-hot encoding - class prediction image index 12

Y_test[12]

array([0., 0., 0., 0., 0., 0., 0., 1., 0., 0.], dtype=float32)

reverse one-hot encoding - images 12

print('Argmax :',np.argmax(Y_test[12]))

Argmax : 7

name of the predicted class

targets_names[np.argmax(Y_test[12])]

'Sneaker'

image predicted

plt.figure(figsize=(6,3))
plt.imshow(X_test[12].reshape(28,28), cmap=plt.cm.binary)
plt.show()

accuracy of the prediction in % - img 12

print('validated at = {t:3.2f} %'.format(t= np.max(predictions[12]*100)))
print('predict Class = ', np.argmax(predictions[12]),' - true Class is = ', np.argmax(Y_test[12]))

validated at = 68.40 %
predict Class =  7  - true Class is =  7

import fonction plotresults - my library

• function - plot_image(i, predictions[i], Y_test, X_test, targets_names)
• function plot_value_array(i, predictions[i], Y_test) #test_labels)

sys.path.append('./my_plot_results')
from my_functions_plot import *

• Correct prediction labels are blue and incorrect prediction labels are red.
• The number gives the percentage for the predicted label.

i = 12
plt.figure(figsize=(6,3))
plt.subplot(1,2,1)
plt.title('img')
plot_image(i, predictions[i], Y_test, X_test, targets_names)
plt.subplot(1,2,2)
plt.title('Class')
plot_value_array(i, predictions[i], Y_test) #test_labels)
plt.show()

Let's plot several images with their predictions. Note that the model can be wrong even when very confident.

# Plot the first X test images, their predicted labels, and the true labels.
# Color correct predictions in blue and incorrect predictions in red.
num_rows = 5
num_cols = 3
num_images = num_rows*num_cols
plt.figure(figsize=(2*2*num_cols, 2*num_rows))
for i in range(num_images):
  plt.subplot(num_rows, 2*num_cols, 2*i+1)
  plot_image(i, predictions[i], Y_test, X_test, targets_names)
  plt.subplot(num_rows, 2*num_cols, 2*i+2)
  plot_value_array(i, predictions[i], Y_test)
plt.tight_layout()
plt.show()

Where Does the Model Make Mistakes ? - Statistics -

• function confusion_matrix from sklearn.metrics
• Compute confusion matrix to evaluate the accuracy of classification.
  
  Explanation: ex: ‘T-shirst/top’
  On the line, the value 768 box (0.0) corresponds to the total number of images detected with a high probability.
  The cumulative sum of this line corresponds to the total number of images of this class in the dataset (10,000).
  The other elements of this line tell us the number of times that another class has mistakenly detected this class ex: case (0,1) = 3, the class 'Trouser' has mistakenly detected the class 'T-shirst /top'.
  
  
• function - mistakes_stat(Y_test, predictions, targets_names, board)

mistakes_stat(Y_test, predictions, targets_names, board=True)

	Label Class	Label Name	Max result	max Class in data	valideted %
0	0	T-shirt/top	828	1000	82.8
1	1	Trouser	962	1000	96.2
2	2	Pullover	813	1000	81.3
3	3	Dress	895	1000	89.5
4	4	Coat	773	1000	77.3
5	5	Sandal	931	1000	93.1
6	6	Shirt	677	1000	67.7
7	7	Sneaker	965	1000	96.5
8	8	Bag	961	1000	96.1
9	9	Ankle boot	944	1000	94.4

Conclusion.

First, this neural network is a basic model. Despite obtaining an accuracy at 99.46% in training and validation at 99.70% without Overfitting. During real tests, this network only succeeds in detecting 83.64% of the 10,000 images injected in the test.

One of the problems comes from the "Flatten" operation which converts images of dimension (28x28) into a vector of (784) pixels. This leads to loss of information on the images.

In image processing in Deep Learning Supervised, we prefer to use a Convolution network.
A.S.A.P - Philippe Jadoul