Deep Learning With Python: Ch. 7

Advanced deep-learning best practices

---

Tópicos

• Técnicas de Programação
  • Funções Callback
  • Programação Orientada a Objetos: Classes e Herança
• API de callback do Keras
• Boas Práticas em Deep Learning
• Aplicações
  • Implementação do Concrete Dropout usando Keras
  • Fitsbook (projeto próprio)

---

Callbacks

1. No Python, toda função é um objeto de primeira ordem

def q(x):
  return x ** 2

def p(x, func):
  print('O quadrado de', x, 'é', func(x))

p(2, q)

O quadrado de 2 é 4

2. Fluxo de um programa

import time

def fit(lr):
  print('Inicio', lr)
  time.sleep(3)
  print('Fim', lr)

lr = 3
fit(lr)
lr = 5
fit(lr)

Inicio 3
Fim 3
Inicio 5
Fim 5

3. Implementação de um callback

def fit(func):
  model = {
      'lr': 100,
      'epoch': 0
  }

  print('Estado Inicial', model)

  for i in range(10):
    lr = func(model)
    time.sleep(2)
    model['epoch'] = model['epoch'] + 1
    print(model)
  
def dynamic_lr(m):
  m['lr'] = m['lr'] / 2.
  
fit(dynamic_lr)

Estado Inicial {'lr': 100, 'epoch': 0}
{'lr': 10.0, 'epoch': 1}
{'lr': 1.0, 'epoch': 2}
{'lr': 0.1, 'epoch': 3}
{'lr': 0.01, 'epoch': 4}
{'lr': 0.001, 'epoch': 5}
{'lr': 0.0001, 'epoch': 6}
{'lr': 1e-05, 'epoch': 7}
{'lr': 1.0000000000000002e-06, 'epoch': 8}
{'lr': 1.0000000000000002e-07, 'epoch': 9}
{'lr': 1.0000000000000002e-08, 'epoch': 10}

Programação Orientada a Objetos

Classes: Visão Geral

Estrutura de dados que associa um conjunto de funções a um estado.

Exemplo: Definindo e criando instâncias de classes

class Bolo:
  def __init__(self):
    self.ingrediente = []
    self.cobertura = ''
    self.assado = False
    self.restante = 1
  
  def assar(self):
    self.assado = True
    print('Bolo Assado')
  
  def colocar_cobertura(self, cobertura):
    self.cobertura = cobertura

  def comer(self):
    if (self.restante >= .5):
      self.restante = self.restante - .5
      print('Hmmmm..')
    else:
      print('O bolo Acabou!!')

bolo1 = Bolo()
bolo1.cobertura = 'chocolate'
bolo1.ingredientes = ['farinha', 'ovo']


bolo1.comer()
print(bolo1.restante)

bolo2 = Bolo()
bolo2.comer()
bolo2.comer()
print(bolo2.restante)

Hmmmm..
0.5
Hmmmm..
Hmmmm..
0.0

Classes: Herança

Uma forma de incorporar métodos e atributos de uma classe para outras classes. Garante a padronização do código.

Exemplo: Herança

class Animal:
  def __init__(self, pes, voa):
    self.pes = pes
    self.voa = voa

  def comer(self):
    print('Comendo...')

  def beber(self):
    print('Bebendo...')

class Cachorro(Animal):
  def __init__(self):
    super(Cachorro, self).__init__(4, False)
  
  def comer(self):
    print('Ração')

  def latir(self):
    print('Au Au')

c = Cachorro()
c.comer()
c.beber()

Ração
Bebendo...

API de Callbacks do Keras

Quando se trabalha com um grande conjunto de dados e uma rede neural complexa treinando por um número alto de épocas, esperar o término da execução dos métodos compile e fit (principalmente) torna-se improdutivo, pelo alto investimento de tempo.

Os callbacks ajudam a ter maior controle sobre o fluxo do programa.

Aplicação 1 - Fitsbook

Link do site

Treinamento de exemplo: identificar digitos do MNIST

%tensorflow_version 1.x
import random
import tensorflow as tf
import numpy as np
from keras import backend as K
import os

from keras import layers
from keras import models
from keras.datasets import mnist
from keras.utils import to_categorical
from keras import callbacks

import matplotlib.pyplot as plt

!pip3 install git+https://github.com/nmcardoso/fitsbook-python
import fitsbook as fb

TensorFlow 1.x selected.

Using TensorFlow backend.

Collecting git+https://github.com/nmcardoso/fitsbook-python
  Cloning https://github.com/nmcardoso/fitsbook-python to /tmp/pip-req-build-aj70s574
  Running command git clone -q https://github.com/nmcardoso/fitsbook-python /tmp/pip-req-build-aj70s574
Requirement already satisfied (use --upgrade to upgrade): fitsbook==0.1 from git+https://github.com/nmcardoso/fitsbook-python in /usr/local/lib/python3.6/dist-packages
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from fitsbook==0.1) (2.21.0)
Requirement already satisfied: keras in /usr/local/lib/python3.6/dist-packages (from fitsbook==0.1) (2.2.5)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from fitsbook==0.1) (1.18.2)
Requirement already satisfied: tensorflow in /tensorflow-1.15.2/python3.6 (from fitsbook==0.1) (1.15.2)
Requirement already satisfied: urllib3<1.25,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->fitsbook==0.1) (1.24.3)
Requirement already satisfied: idna<2.9,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->fitsbook==0.1) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->fitsbook==0.1) (2019.11.28)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->fitsbook==0.1) (3.0.4)
Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from keras->fitsbook==0.1) (1.1.0)
Requirement already satisfied: keras-applications>=1.0.8 in /usr/local/lib/python3.6/dist-packages (from keras->fitsbook==0.1) (1.0.8)
Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.6/dist-packages (from keras->fitsbook==0.1) (1.4.1)
Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from keras->fitsbook==0.1) (2.10.0)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.6/dist-packages (from keras->fitsbook==0.1) (1.12.0)
Requirement already satisfied: pyyaml in /usr/local/lib/python3.6/dist-packages (from keras->fitsbook==0.1) (3.13)
Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (3.10.0)
Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (1.12.1)
Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (1.27.2)
Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (3.2.0)
Requirement already satisfied: tensorflow-estimator==1.15.1 in /tensorflow-1.15.2/python3.6 (from tensorflow->fitsbook==0.1) (1.15.1)
Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (1.1.0)
Requirement already satisfied: gast==0.2.2 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (0.2.2)
Requirement already satisfied: tensorboard<1.16.0,>=1.15.0 in /tensorflow-1.15.2/python3.6 (from tensorflow->fitsbook==0.1) (1.15.0)
Requirement already satisfied: wheel>=0.26; python_version >= "3" in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (0.34.2)
Requirement already satisfied: astor>=0.6.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (0.8.1)
Requirement already satisfied: google-pasta>=0.1.6 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (0.2.0)
Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.6/dist-packages (from tensorflow->fitsbook==0.1) (0.9.0)
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf>=3.6.1->tensorflow->fitsbook==0.1) (46.0.0)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow->fitsbook==0.1) (1.0.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.6/dist-packages (from tensorboard<1.16.0,>=1.15.0->tensorflow->fitsbook==0.1) (3.2.1)
Building wheels for collected packages: fitsbook
  Building wheel for fitsbook (setup.py) ... done
  Created wheel for fitsbook: filename=fitsbook-0.1-cp36-none-any.whl size=2181 sha256=b8f32bd3916cc03055d575bc972903f24864698f9ec0e67157f31c95bb4cb22f
  Stored in directory: /tmp/pip-ephem-wheel-cache-hd0xdmv2/wheels/99/2a/a0/b37947be424acac83a373b37b38a2bfaa4557b049c19426655
Successfully built fitsbook

dense_model = models.Sequential()
dense_model.add(layers.Dense(512, activation='relu', input_shape=(28 * 28,)))
dense_model.add(layers.Dense(10, activation='softmax'))
dense_model.compile(optimizer='rmsprop',
                    loss='categorical_crossentropy',
                    metrics=['accuracy'])

dense_model.summary()

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:541: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4432: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:793: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3576: The name tf.log is deprecated. Please use tf.math.log instead.

Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 512)               401920    
_________________________________________________________________
dense_2 (Dense)              (None, 10)                5130      
=================================================================
Total params: 407,050
Trainable params: 407,050
Non-trainable params: 0
_________________________________________________________________

(dense_train_img, dense_train_lbl), (dense_test_img, dense_test_lbl) = mnist.load_data()
dense_train_img = dense_train_img.reshape((60000, 28 * 28))
dense_train_img = dense_train_img.astype('float32') / 255

dense_test_img = dense_test_img.reshape((10000, 28 * 28))
dense_test_img = dense_test_img.astype('float32') / 255

dense_train_lbl = to_categorical(dense_train_lbl)
dense_test_lbl = to_categorical(dense_test_lbl)

Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz
11493376/11490434 [==============================] - 0s 0us/step

%%time
dense_history = dense_model.fit(dense_train_img,
                                dense_train_lbl, 
                                epochs=10, 
                                batch_size=128,
                                validation_split=0.17,
                                callbacks=[fb.callbacks.FitsbookCallback()],
                                validation_data=(dense_test_img, dense_test_lbl))

WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1033: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1020: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3005: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

Train on 60000 samples, validate on 10000 samples
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:197: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:207: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:216: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:223: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

[Fitsbook]: Monitoring this training in real time https://natan.ninja/#/stats/20
Epoch 1/10
60000/60000 [==============================] - 6s 94us/step - loss: 0.2598 - acc: 0.9253 - val_loss: 0.1285 - val_acc: 0.9610
Epoch 2/10
60000/60000 [==============================] - 5s 90us/step - loss: 0.1074 - acc: 0.9684 - val_loss: 0.1120 - val_acc: 0.9646
Epoch 3/10
60000/60000 [==============================] - 5s 88us/step - loss: 0.0717 - acc: 0.9786 - val_loss: 0.0883 - val_acc: 0.9726
Epoch 4/10
60000/60000 [==============================] - 5s 88us/step - loss: 0.0528 - acc: 0.9846 - val_loss: 0.0699 - val_acc: 0.9787
Epoch 5/10
60000/60000 [==============================] - 5s 87us/step - loss: 0.0394 - acc: 0.9879 - val_loss: 0.0661 - val_acc: 0.9800
Epoch 6/10
60000/60000 [==============================] - 5s 90us/step - loss: 0.0312 - acc: 0.9909 - val_loss: 0.0657 - val_acc: 0.9806
Epoch 7/10
60000/60000 [==============================] - 5s 91us/step - loss: 0.0231 - acc: 0.9934 - val_loss: 0.0615 - val_acc: 0.9815
Epoch 8/10
60000/60000 [==============================] - 5s 87us/step - loss: 0.0187 - acc: 0.9940 - val_loss: 0.0635 - val_acc: 0.9826
Epoch 9/10
60000/60000 [==============================] - 5s 85us/step - loss: 0.0146 - acc: 0.9956 - val_loss: 0.0775 - val_acc: 0.9792
Epoch 10/10
60000/60000 [==============================] - 5s 86us/step - loss: 0.0114 - acc: 0.9966 - val_loss: 0.0703 - val_acc: 0.9825
[Fitsbook]: Training ended successfuly. https://natan.ninja/#/stats/20
CPU times: user 1min 22s, sys: 3.81 s, total: 1min 26s
Wall time: 57.3 s

def plot_figure(history):
  epochs = history.epoch
  plt.figure(figsize=(10, 6))
  plt.plot(epochs, history.history['acc'], linestyle='--', color='blue', label='Train Acc')
  plt.plot(epochs, history.history['val_acc'], color='blue', label='Validation Acc')
  plt.plot(epochs, history.history['loss'], linestyle='--', color='green', label='Train Loss')
  plt.plot(epochs, history.history['val_loss'], color='green', label='Validation Loss')
  plt.grid(b=True)
  plt.title('Dense Layers Metrics', fontsize=17)
  plt.ylabel('Metrics', fontsize=13)
  plt.xlabel('Epoch', fontsize=13)
  plt.legend(fontsize='large')
  plt.show()
plot_figure(dense_history)

Eventos

• on_batch_begin
• on_batch_end
• on_epoch_begin
• on_epoch_end
• on_train_begin
• on_train_end

Implementações

• Keras - Módulo Callback
• Keras - Função train-loop
• FitsbookPython - Classe Callback

Aplicação 2 - Concrete Dropout

• Artigo teórico
• Artigo com implementação
• Keras - Classe Wrapper

Deep-learning best practices

1. Batch Normalization

Normalizar seus dados de entrada para um intervalo e centrar os dados em zero.

2. Depthwise separable convolution

Se as localizações da imagem estão correlacionadas, mas os canais puderem ser interpretados idependentemente, faz sentido separar as camadas de convolução por canal e juntá-las depois.

3. Hyperparameter Optimization

Quando se desenvolve uma projeta uma rede neural, muitas decisões arbitrárias são tomadas. Quantas camadas empilhar? Quantas unidades ou filtros devem ir em cada camada? Devo usar relu ou outra função de ativação? Quanto dropout devo usar?

Na prática, uma boa intuição de otimização de hiperparâmetros é construída através de várias experiências no decorrer do tempo. As decisões iniciais serão sempre subotimizadas até que se desenvolva uma boa intuição de como ajustar os hiperparâmetros.

As ferramentas abaixo otimizam os hiperparâmetros automaticamente

• Hyperopt - https://github.com/hyperopt/hyperopt
• Hyperas - https://github.com/maxpumperla/hyperas

4. Model Ensembling

Compor a predição final usando predições parciais de vários modelos.

preds_a = model_a.predict(x_val)
preds_b = model_b.predict(x_val)
preds_c = model_c.predict(x_val)
preds_d = model_d.predict(x_val)
final_preds = 0.5 * preds_a + 0.25 * preds_b + 0.1 * preds_c + 0.15 * preds_d