Train donkeycar with GPU support in Colab

• Credit
• Step 1: Create environment
  • Git Clone the donkeycar repository
  • Install donkey car
  • Create Project
• Step 2: Supply Data
  • Step 2 opt A: Supply own data
    • Zip data on raspi
    • Copy the zipped tub to your local PC
    • Upload the tub from Google Drive
    • Check quality of data
  • Supply GitHub hosted data
  • Step 2 opt B: Use data from RoboCarEsslingen
    • Check quality of data
  • Use data from Connected Autonomous Mobility
    • Check quality of data
• Step 3: Upload myconfig.py
  • Get myconfig.py
• Step 4: Train your model
  • Step 4 opt A: Transfer learning using pre-trained model
    • Upload pre-trained model
      • Start transfer learning of pre-trained model
  • Step 4 opt B: Train RNN model
• Step 5: Check model and transfer data
  • Plot loss curve of model
  • Plot commands and predictions
  • Copy the trained model back to Donkey Car (Pi)
  • Copy the file from your PC or Mac to the Raspberry Pi using Filezilla or scp command.
  • Start Autopilot on Pi
• Step 6: Bonus - Salient Object Visualization

Credit

Note: This notebook is based on https://colab.research.google.com/github/robocarstore/donkey-car-training-on-google-colab/blob/master/Donkey_Car_Training_using_Google_Colab.ipynb

Get necessary libraries

#collapse-hide
%tensorflow_version 1.13.1
import tensorflow
from google.colab import drive
from google.colab import files
from IPython.display import Image
import glob
import shutil 

Step 1: Create environment

To train a neural network for the donkeycar we need a few components

• install donkeycar
• upload data via
  • direct upload
  • mount Google drive

Note: Donkeycar at the time of writing in March 2020 uses Tensorflow 1.13, therefore version 1.xx is installed

#collapse-show 

print(tensorflow.__version__)

Git Clone the donkeycar repository

Get the latest donkeycar version from GitHub

Note: The default branch is "dev", however, the documentation is for the master branch.

!git clone https://github.com/autorope/donkeycar.git 
%cd /content/donkeycar

!git checkout master

Cloning into 'donkeycar'...
remote: Enumerating objects: 12972, done.
remote: Total 12972 (delta 0), reused 0 (delta 0), pack-reused 12972
Receiving objects: 100% (12972/12972), 67.74 MiB | 31.90 MiB/s, done.
Resolving deltas: 100% (8193/8193), done.
/content/donkeycar
Branch 'master' set up to track remote branch 'master' from 'origin'.
Switched to a new branch 'master'

Install donkey car

Different to the description at http://docs.donkeycar.com/guide/host_pc/setup_ubuntu/ we create no anaconda environment since the script is supposed to run on Colab which will delete the instance anyway once you disconnect the notebook.

!pip3 install -e .[pc]

Obtaining file:///content/donkeycar
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (1.18.4)
Requirement already satisfied: pillow in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (7.0.0)
Requirement already satisfied: docopt in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (0.6.2)
Requirement already satisfied: tornado in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (4.5.3)
Requirement already satisfied: requests in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (2.23.0)
Requirement already satisfied: h5py in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (2.10.0)
Requirement already satisfied: moviepy in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (0.2.3.5)
Requirement already satisfied: pandas in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (1.0.3)
Requirement already satisfied: PrettyTable in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (0.7.2)
Collecting paho-mqtt
  Downloading https://files.pythonhosted.org/packages/59/11/1dd5c70f0f27a88a3a05772cd95f6087ac479fac66d9c7752ee5e16ddbbc/paho-mqtt-1.5.0.tar.gz (99kB)
     |████████████████████████████████| 102kB 3.4MB/s 
Requirement already satisfied: matplotlib in /usr/local/lib/python3.6/dist-packages (from donkeycar==3.1.2) (3.2.1)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.6/dist-packages (from requests->donkeycar==3.1.2) (1.24.3)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.6/dist-packages (from requests->donkeycar==3.1.2) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.6/dist-packages (from requests->donkeycar==3.1.2) (2020.4.5.1)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.6/dist-packages (from requests->donkeycar==3.1.2) (2.9)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from h5py->donkeycar==3.1.2) (1.12.0)
Requirement already satisfied: imageio<3.0,>=2.1.2 in /usr/local/lib/python3.6/dist-packages (from moviepy->donkeycar==3.1.2) (2.4.1)
Requirement already satisfied: tqdm<5.0,>=4.11.2 in /usr/local/lib/python3.6/dist-packages (from moviepy->donkeycar==3.1.2) (4.41.1)
Requirement already satisfied: decorator<5.0,>=4.0.2 in /usr/local/lib/python3.6/dist-packages (from moviepy->donkeycar==3.1.2) (4.4.2)
Requirement already satisfied: pytz>=2017.2 in /usr/local/lib/python3.6/dist-packages (from pandas->donkeycar==3.1.2) (2018.9)
Requirement already satisfied: python-dateutil>=2.6.1 in /usr/local/lib/python3.6/dist-packages (from pandas->donkeycar==3.1.2) (2.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->donkeycar==3.1.2) (1.2.0)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib->donkeycar==3.1.2) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib->donkeycar==3.1.2) (2.4.7)
Building wheels for collected packages: paho-mqtt
  Building wheel for paho-mqtt (setup.py) ... done
  Created wheel for paho-mqtt: filename=paho_mqtt-1.5.0-cp36-none-any.whl size=61416 sha256=361839e2e72f674579aacf767c1756b47c4de8dcf28e6e77ecc0f1d13b6a6727
  Stored in directory: /root/.cache/pip/wheels/02/94/6c/8474137cb7a5a3e001d70a22c8ff919caee69435376bccce79
Successfully built paho-mqtt
Installing collected packages: paho-mqtt, donkeycar
  Running setup.py develop for donkeycar
Successfully installed donkeycar paho-mqtt-1.5.0

Create Project

In this step the following actions take place

• create necessary folders (models, data, logs)
• copying necessary files into folders (manage.py, myconfig.py etc.)

!donkey createcar --path /content/mycar

using donkey v3.1.2 ...
Creating car folder: /content/mycar
making dir  /content/mycar
Creating data & model folders.
making dir  /content/mycar/models
making dir  /content/mycar/data
making dir  /content/mycar/logs
Copying car application template: complete
Copying car config defaults. Adjust these before starting your car.
Copying train script. Adjust these before starting your car.
Copying my car config overrides
Donkey setup complete.

Step 2: Supply Data

In order to train the neural network we need to supply trainings data which are recorded on the raspi during driving the donkeycar on the track

Step 2 opt A: Supply own data

If you have own data proceed here, if you want to use data which were made available via GitHub please continue to (link only works in blog not in Colab) section Supply GitHub hosted data

Zip data on raspi

Copy the following code and run on raspi

Note: Copying of the data is much faster if the data is zipped to one file.

cd ~/mycar/data
# either compress just one folder
tar -czf tub_xx_yyyy_mm_dd.tar.gz tub_xx_yyyy_mm_dd
# or all folders starting with "tub"
tar -czf trainingsData2020_03-01.tar.gz tub*

This will create a tub_xx_yyyy_mm_dd.tar.gz file under ~/mycar/data

Copy the zipped tub to your local PC

Run this on your local pc if you are using linux/mac

sftp pi@raspberry.local
cd ~/mycar/data
get tub_xx_yyyy_mm_dd.tar.gz

If you are on a windows, download sftp utility like filezilla or putty

Define your tub name here

tub_name="tubVaihingenIIICleaned200126"

Upload the tub from Google Drive

First upload the tub_x_yyyy_mm_dd.tar.gz to Google Drive. We will then mount Google Drive from colab and copy the data from Drive directly.

Note: To copy data from Google Drive to Colab is faster than uploading it from local machine.

When you run the cell below, you will need to click the link and generate an authorization code to for colab to access your drive.

drive.mount('/content/drive')

Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

Enter your authorization code:
··········
Mounted at /content/drive

Suppose you upload the tub_xx_yyyy_mm_dd.tar.gz to Google Drive/mycar/tub_xx_yyyy_mm_dd.tar.gz, this is how you copy it from Google Drive to colab

%cd /content/mycar
!rm -rf data
!mkdir data
%cd /content/mycar/data
!cp /content/drive/My\ Drive/myCar/{tub_name}.tar.gz .

/content/mycar
/content/mycar/data

And untar it to the right place

%cd /content/mycar/data
!tar -xzf {tub_name}.tar.gz

/content/mycar/data

Lets look at one image to see we got valid data

%cd /content/mycar/data/tubVaihingenIIICleaned200126/
file = glob.glob("*.jpg")
Image(file[100])

/content/mycar/data/tubVaihingenIIICleaned200126

Check quality of data

You want data which has left and right turns preferably in equal shares. A histogram is a good tool to check if this is the case. You can use a donkeycar tool for that

donkey tubhist <tub_path> --rec=<"user/angle">

The histogram shows that mainly the car drove straight ahead and more left turns than right turns. It is good practice to drive a course clock wise and anti clock wise to avoid this imbalance.

%cd /content/mycar
!donkey tubhist --tub data/{tub_name} --rec="user/angle"
file = glob.glob("tubVaihingenIIICleaned200126_hist_user_angle.png")
Image(file[0])

/content/mycar
using donkey v3.1.2 ...
TubGroup:tubpaths: ['/content/mycar/data/tubVaihingenIIICleaned200126']
joining the tubs 14234 records together. This could take 0 minutes.
saving image to: tubVaihingenIIICleaned200126_hist_user_angle.png
<Figure size 640x480 with 1 Axes>

Next step is to train your model in section Upload local files (Link only works in blog not in Colab)

Supply GitHub hosted data

If you don't have own training data you might want to use an example data set

Note: The training data is cleaned (tubclean) but whether or not you get a good working model out of it...

See instructions below how to get the data into the Colab environment

The training data are hosted on GitHub

• clone GitHub repo
• move file to data folder
• unzip file

Step 2 opt B: Use data from RoboCarEsslingen

The first data set we use is from RoboCar Esslingen GitHub which is operated by the meetup group Esslinger Makerspace Projekt: Autonomen RoboCar bauen located in Esslingen, Germany.
The data was recorded at #8.7 Quarterly Hack: DIYrobocars Build & Race in Suttgart with kind support of Bosch.
If you want to use data from Connected Atonomous Mobility proceed in chapter (link only works in blog not in Colab) section Use data from Connected Autonomous Mobility

%cd /content/mycar
!rm -rf data
!mkdir data
%cd /content/mycar/data
!pwd
! git clone --recursive https://github.com/RoboCarEsslingen/trainingData.git

Move zip file to data folder and unzip

shutil.move("/content/mycar/data/trainingData/tubVaihingenIIICleaned200126.tar.gz", "/content/mycar/data") 

Unzip the training data file

%cd /content/mycar/data
!tar -xzf tubVaihingenIIICleaned200126.tar.gz

Check quality of data

You want data which has left and right turns preferably in equal shares. A histogram is a good tool to check if this is the case. You can use a donkeycar tool for that

donkey tubhist <tub_path> --rec=<"user/angle">

The histogram shows that mainly the car drove straight ahead and more left turns than right turns. It is good practice to drive a course clock wise and anti clock wise to avoid this imbalance.

%cd /content/mycar
!donkey tubhist --tub data/{tub_name} --rec="user/angle"
file = glob.glob("tubVaihingenIIICleaned200126_hist_user_angle.png")
Image(file[0])

Use data from Connected Autonomous Mobility

Another data set is from Connected Autonomous Mobility We clone the whole repo because I don't know how to download a single file from a GitHub repo, if you know how to this than please leave a note the comment section.

%cd /content/mycar
!rm -rf data
!mkdir data
%cd /content/mycar/data
!pwd
! git clone --recursive https://github.com/connected-autonomous-mobility/20-data

Move zip file to data folder and unzip

import shutil 
shutil.move("/content/mycar/data/20-data/20190414-BOSCH-Solaris-Course/tub_36_19-04-13.zip", "/content/mycar/data") 

Unzip the training data file

%cd /content/mycar/data
!unzip tub_36_19-04-13.zip

Lets look at one image to get an impression what the car saw.

# This is formatted as code

%cd /content/mycar/data/tub_36_19-04-13/
file = glob.glob("*.jpg")
Image(file[300])

Check quality of data

You want data which has left and right turns preferably in equal shares. A histogram is a good tool to check if this is the case. You can use a donkeycar tool for that

donkey tubhist <tub_path> --rec=<"user/angle">

The histogram shows that mainly the car drove straight ahead, left and right turns are pretty well balanced

tub_name="tub_36_19-04-13"
%cd /content/mycar
!donkey tubhist --tub data/{tub_name} --rec="user/angle"
file = glob.glob("tub_36_19-04-13_hist_user_angle.png")
Image(file[0])

Step 3: Upload myconfig.py

You can upload files from local machine as well, but probably is slower than above approach downloading files from Google Drive

# uploaded = files.upload()

Get myconfig.py

The file myconfig.py has to be the identical during training and driving, therefore it makes sense to upload the myconfig.py which you are using on the car.

Note: In myconfig.py there are parameters which control the training such as:

 line parameter --type to the python manage.py train and drive commands.

 DEFAULT_MODEL_TYPE = 'linear'   (linear|categorical|rnn|imu|behavior|3d|localizer|latent)

 BATCH_SIZE = 128                how many records to use when doing one pass of gradient decent. Use a smaller number if your gpu is running out of memory.

 TRAIN_TEST_SPLIT = 0.8          what percent of records to use for training. the remaining used for validation.

 MAX_EPOCHS = 100                how many times to visit all records of your data

 SHOW_PLOT = True                would you like to see a pop up display of final loss?

 VEBOSE_TRAIN = True             would you like to see a progress bar with text during training?

 USE_EARLY_STOP = True           would you like to stop the training if we see it's not improving fit?

 EARLY_STOP_PATIENCE = 5         how many epochs to wait before no improvement

 MIN_DELTA = .0005               early stop will want this much loss change before calling it improved.

 PRINT_MODEL_SUMMARY = True      print layers and weights to stdout

 OPTIMIZER = None                adam, sgd, rmsprop, etc.. None accepts default

 LEARNING_RATE = 0.001           only used when OPTIMIZER specified

 LEARNING_RATE_DECAY = 0.0       only used when OPTIMIZER specified

 SEND_BEST_MODEL_TO_PI = False   change to true to automatically send best model during training

 CACHE_IMAGES = True             keep images in memory. will speed successive epochs, but crater if not enough mem.



 PRUNE_CNN = False               This will remove weights from your model. The primary goal is to increase performance.

 PRUNE_PERCENT_TARGET = 75        The desired percentage of pruning.

 PRUNE_PERCENT_PER_ITERATION = 20  Percentage of pruning that is perform per iteration.

 PRUNE_VAL_LOSS_DEGRADATION_LIMIT = 0.2  The max amount of validation loss that is permitted during pruning.

 PRUNE_EVAL_PERCENT_OF_DATASET = .05   percent of dataset used to perform evaluation of model.
RNN or 3D  
SEQUENCE_LENGTH = 3             #some models use a number of images over time. This controls how many.  

# # Region of interest cropping
# # only supported in Categorical and Linear models.
ROI_CROP_TOP = 0                    #the number of rows of pixels to ignore on the top of the image  
 ROI_CROP_BOTTOM = 0            #the number of rows of pixels to ignore on the bottom of the image

%cd /content/mycar
!cp /content/drive/My\ Drive/myCar/myconfig.py .

/content/mycar

Step 4: Train your model

There are several types of modes available:

• linear
• categorical
• rnn
• imu
• behavior
• 3d
• localizer
  And you can use pre-trained models by adding a flag
  • [--transfer=<model>]

Step 4 opt A: Transfer learning using pre-trained model

Note: You can use a pre-trained model and use transfer learning

Do not forget to set the variables in myconfig.py

FREEZE_LAYERS = True  
`#default False will allow all layers to be modified by training

NUM_LAST_LAYERS_TO_TRAIN = 7  
`#when freezing layers, how many layers from the last should be allowed to train?

Upload pre-trained model

Upload model in case you want to use a pre-trained model for transfer learning. To define which layers shall be trained and which shall be frozen set the parameters in `myconfig.py``

Model transfer options

When copying weights during a model transfer operation, should we freeze a certain number of layers to the incoming weights and not allow them to change during training?

FREEZE_LAYERS = False               #default False will allow all layers to be modified by training
NUM_LAST_LAYERS_TO_TRAIN = 7        #when freezing layers, how many layers from the last should be allowed to train?

%cd /content/mycar/models
!cp /content/drive/My\ Drive/myCar/base_linear.h5 .

/content/mycar/models

Plot the model structure

from tensorflow.keras.utils import plot_model
from tensorflow.keras.models import load_model
%cd /content/mycar/models
model = load_model('base_linear.h5')
plot_model(
    model,
    to_file="model.png",
    show_shapes=False,
    show_layer_names=True,
    rankdir="TB",
    expand_nested=False,
    dpi=96,
)

/content/mycar/models
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.

Start transfer learning of pre-trained model

Use the manage.py script to start training

!python /content/mycar/manage.py train  --type=linear --transfer=/content/mycar/models/base_linear.h5  --model=/content/mycar/models/mypilot.h5

using donkey v3.1.2 ...
loading config file: /content/mycar/config.py
myconfig myconfig.py
loading personal config over-rides from myconfig.py

config loaded
"get_model_by_type" model Type is: linear
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
training with model type <class 'donkeycar.parts.keras.KerasLinear'>
loading weights from model /content/mycar/models/base_linear.h5
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
2020-05-28 15:11:01.559596: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-28 15:11:01.562497: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.563191: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:04.0
2020-05-28 15:11:01.563474: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-28 15:11:01.565174: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-28 15:11:01.566931: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-28 15:11:01.567244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-28 15:11:01.568968: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-28 15:11:01.570096: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-28 15:11:01.574256: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-28 15:11:01.574380: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.575112: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.575708: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-05-28 15:11:01.581927: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2020-05-28 15:11:01.582168: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x67cee00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-28 15:11:01.582199: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-28 15:11:01.673210: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.673930: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x67cefc0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-28 15:11:01.673963: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2020-05-28 15:11:01.674127: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.674884: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:04.0
2020-05-28 15:11:01.674977: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-28 15:11:01.675006: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-28 15:11:01.675028: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-28 15:11:01.675051: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-28 15:11:01.675070: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-28 15:11:01.675089: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-28 15:11:01.675109: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-28 15:11:01.675183: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.675764: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.676337: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-05-28 15:11:01.676400: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-28 15:11:01.677577: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-28 15:11:01.677605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-05-28 15:11:01.677616: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-05-28 15:11:01.677718: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.678342: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:11:01.678954: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-28 15:11:01.678998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14858 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)
Model: "model"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
img_in (InputLayer)             [(None, 120, 160, 3) 0                                            
__________________________________________________________________________________________________
cropping2d (Cropping2D)         (None, 120, 160, 3)  0           img_in[0][0]                     
__________________________________________________________________________________________________
batch_normalization_v1 (BatchNo (None, 120, 160, 3)  12          cropping2d[0][0]                 
__________________________________________________________________________________________________
conv2d_1 (Conv2D)               (None, 58, 78, 24)   1824        batch_normalization_v1[0][0]     
__________________________________________________________________________________________________
dropout (Dropout)               (None, 58, 78, 24)   0           conv2d_1[0][0]                   
__________________________________________________________________________________________________
conv2d_2 (Conv2D)               (None, 27, 37, 32)   19232       dropout[0][0]                    
__________________________________________________________________________________________________
dropout_1 (Dropout)             (None, 27, 37, 32)   0           conv2d_2[0][0]                   
__________________________________________________________________________________________________
conv2d_3 (Conv2D)               (None, 12, 17, 64)   51264       dropout_1[0][0]                  
__________________________________________________________________________________________________
dropout_2 (Dropout)             (None, 12, 17, 64)   0           conv2d_3[0][0]                   
__________________________________________________________________________________________________
conv2d_4 (Conv2D)               (None, 10, 15, 64)   36928       dropout_2[0][0]                  
__________________________________________________________________________________________________
dropout_3 (Dropout)             (None, 10, 15, 64)   0           conv2d_4[0][0]                   
__________________________________________________________________________________________________
conv2d_5 (Conv2D)               (None, 8, 13, 64)    36928       dropout_3[0][0]                  
__________________________________________________________________________________________________
dropout_4 (Dropout)             (None, 8, 13, 64)    0           conv2d_5[0][0]                   
__________________________________________________________________________________________________
flattened (Flatten)             (None, 6656)         0           dropout_4[0][0]                  
__________________________________________________________________________________________________
dense (Dense)                   (None, 100)          665700      flattened[0][0]                  
__________________________________________________________________________________________________
dropout_5 (Dropout)             (None, 100)          0           dense[0][0]                      
__________________________________________________________________________________________________
dense_1 (Dense)                 (None, 50)           5050        dropout_5[0][0]                  
__________________________________________________________________________________________________
dropout_6 (Dropout)             (None, 50)           0           dense_1[0][0]                    
__________________________________________________________________________________________________
n_outputs0 (Dense)              (None, 1)            51          dropout_6[0][0]                  
__________________________________________________________________________________________________
n_outputs1 (Dense)              (None, 1)            51          dropout_6[0][0]                  
==================================================================================================
Total params: 817,040
Trainable params: 817,034
Non-trainable params: 6
__________________________________________________________________________________________________
None
found 0 pickles writing json records and images in tub /content/mycar/data/tubVaihingenIIICleaned200126
/content/mycar/data/tubVaihingenIIICleaned200126
collating 14234 records ...
train: 11387, val: 2847
total records: 14234
steps_per_epoch 177
Epoch 1/100
2020-05-28 15:11:04.890586: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-28 15:11:06.394756: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
176/177 [============================>.] - ETA: 0s - loss: 0.3672 - n_outputs0_loss: 0.2321 - n_outputs1_loss: 0.1352Epoch 1/100
 44/177 [======>.......................] - ETA: 6s - loss: 0.6511 - n_outputs0_loss: 0.6171 - n_outputs1_loss: 0.0341
Epoch 00001: val_loss improved from inf to 0.65115, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 19s 106ms/step - loss: 0.3660 - n_outputs0_loss: 0.2315 - n_outputs1_loss: 0.1345 - val_loss: 0.6511 - val_n_outputs0_loss: 0.6171 - val_n_outputs1_loss: 0.0341
Epoch 2/100
175/177 [============================>.] - ETA: 0s - loss: 0.1948 - n_outputs0_loss: 0.1780 - n_outputs1_loss: 0.0169Epoch 1/100
 40/177 [=====>........................] - ETA: 1s - loss: 0.6370 - n_outputs0_loss: 0.6252 - n_outputs1_loss: 0.0119
Epoch 00002: val_loss improved from 0.65115 to 0.63501, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 27ms/step - loss: 0.1954 - n_outputs0_loss: 0.1785 - n_outputs1_loss: 0.0168 - val_loss: 0.6350 - val_n_outputs0_loss: 0.6229 - val_n_outputs1_loss: 0.0122
Epoch 3/100
175/177 [============================>.] - ETA: 0s - loss: 0.1780 - n_outputs0_loss: 0.1649 - n_outputs1_loss: 0.0131Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.6386 - n_outputs0_loss: 0.6316 - n_outputs1_loss: 0.0070
Epoch 00003: val_loss did not improve from 0.63501
177/177 [==============================] - 4s 25ms/step - loss: 0.1778 - n_outputs0_loss: 0.1647 - n_outputs1_loss: 0.0131 - val_loss: 0.6390 - val_n_outputs0_loss: 0.6322 - val_n_outputs1_loss: 0.0068
Epoch 4/100
174/177 [============================>.] - ETA: 0s - loss: 0.1637 - n_outputs0_loss: 0.1516 - n_outputs1_loss: 0.0120Epoch 1/100
 44/177 [======>.......................] - ETA: 1s - loss: 0.6309 - n_outputs0_loss: 0.6175 - n_outputs1_loss: 0.0133
Epoch 00004: val_loss improved from 0.63501 to 0.63087, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 26ms/step - loss: 0.1635 - n_outputs0_loss: 0.1515 - n_outputs1_loss: 0.0120 - val_loss: 0.6309 - val_n_outputs0_loss: 0.6175 - val_n_outputs1_loss: 0.0133
Epoch 5/100
175/177 [============================>.] - ETA: 0s - loss: 0.1544 - n_outputs0_loss: 0.1429 - n_outputs1_loss: 0.0115Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.3351 - n_outputs0_loss: 0.3270 - n_outputs1_loss: 0.0081
Epoch 00005: val_loss improved from 0.63087 to 0.34012, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 27ms/step - loss: 0.1543 - n_outputs0_loss: 0.1428 - n_outputs1_loss: 0.0115 - val_loss: 0.3401 - val_n_outputs0_loss: 0.3315 - val_n_outputs1_loss: 0.0086
Epoch 6/100
175/177 [============================>.] - ETA: 0s - loss: 0.1488 - n_outputs0_loss: 0.1371 - n_outputs1_loss: 0.0117Epoch 1/100
 43/177 [======>.......................] - ETA: 1s - loss: 0.1704 - n_outputs0_loss: 0.1677 - n_outputs1_loss: 0.0027
Epoch 00006: val_loss improved from 0.34012 to 0.16988, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 26ms/step - loss: 0.1489 - n_outputs0_loss: 0.1372 - n_outputs1_loss: 0.0117 - val_loss: 0.1699 - val_n_outputs0_loss: 0.1673 - val_n_outputs1_loss: 0.0026
Epoch 7/100
174/177 [============================>.] - ETA: 0s - loss: 0.1403 - n_outputs0_loss: 0.1290 - n_outputs1_loss: 0.0113Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.1439 - n_outputs0_loss: 0.1418 - n_outputs1_loss: 0.0021
Epoch 00007: val_loss improved from 0.16988 to 0.14487, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 26ms/step - loss: 0.1403 - n_outputs0_loss: 0.1290 - n_outputs1_loss: 0.0113 - val_loss: 0.1449 - val_n_outputs0_loss: 0.1429 - val_n_outputs1_loss: 0.0020
Epoch 8/100
174/177 [============================>.] - ETA: 0s - loss: 0.1326 - n_outputs0_loss: 0.1223 - n_outputs1_loss: 0.0103Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.1371 - n_outputs0_loss: 0.1334 - n_outputs1_loss: 0.0037
Epoch 00008: val_loss improved from 0.14487 to 0.13729, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 26ms/step - loss: 0.1325 - n_outputs0_loss: 0.1221 - n_outputs1_loss: 0.0104 - val_loss: 0.1373 - val_n_outputs0_loss: 0.1337 - val_n_outputs1_loss: 0.0036
Epoch 9/100
175/177 [============================>.] - ETA: 0s - loss: 0.1266 - n_outputs0_loss: 0.1164 - n_outputs1_loss: 0.0102Epoch 1/100
 41/177 [=====>........................] - ETA: 1s - loss: 0.1297 - n_outputs0_loss: 0.1279 - n_outputs1_loss: 0.0018
Epoch 00009: val_loss improved from 0.13729 to 0.12960, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 27ms/step - loss: 0.1267 - n_outputs0_loss: 0.1163 - n_outputs1_loss: 0.0103 - val_loss: 0.1296 - val_n_outputs0_loss: 0.1277 - val_n_outputs1_loss: 0.0018
Epoch 10/100
175/177 [============================>.] - ETA: 0s - loss: 0.1211 - n_outputs0_loss: 0.1109 - n_outputs1_loss: 0.0101Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.1316 - n_outputs0_loss: 0.1299 - n_outputs1_loss: 0.0017    
Epoch 00010: val_loss did not improve from 0.12960
177/177 [==============================] - 4s 25ms/step - loss: 0.1208 - n_outputs0_loss: 0.1107 - n_outputs1_loss: 0.0101 - val_loss: 0.1326 - val_n_outputs0_loss: 0.1309 - val_n_outputs1_loss: 0.0017
Epoch 11/100
175/177 [============================>.] - ETA: 0s - loss: 0.1195 - n_outputs0_loss: 0.1088 - n_outputs1_loss: 0.0107Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.1345 - n_outputs0_loss: 0.1311 - n_outputs1_loss: 0.0035
Epoch 00011: val_loss did not improve from 0.12960
177/177 [==============================] - 4s 25ms/step - loss: 0.1195 - n_outputs0_loss: 0.1087 - n_outputs1_loss: 0.0107 - val_loss: 0.1341 - val_n_outputs0_loss: 0.1308 - val_n_outputs1_loss: 0.0034
Epoch 12/100
175/177 [============================>.] - ETA: 0s - loss: 0.1116 - n_outputs0_loss: 0.1021 - n_outputs1_loss: 0.0095Epoch 1/100
 39/177 [=====>........................] - ETA: 1s - loss: 0.1315 - n_outputs0_loss: 0.1284 - n_outputs1_loss: 0.0031
Epoch 00012: val_loss improved from 0.12960 to 0.12898, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 26ms/step - loss: 0.1120 - n_outputs0_loss: 0.1025 - n_outputs1_loss: 0.0095 - val_loss: 0.1290 - val_n_outputs0_loss: 0.1260 - val_n_outputs1_loss: 0.0029
Epoch 13/100
175/177 [============================>.] - ETA: 0s - loss: 0.1108 - n_outputs0_loss: 0.1016 - n_outputs1_loss: 0.0092Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.1257 - n_outputs0_loss: 0.1231 - n_outputs1_loss: 0.0026
Epoch 00013: val_loss improved from 0.12898 to 0.12570, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 26ms/step - loss: 0.1106 - n_outputs0_loss: 0.1014 - n_outputs1_loss: 0.0092 - val_loss: 0.1257 - val_n_outputs0_loss: 0.1232 - val_n_outputs1_loss: 0.0025
Epoch 14/100
175/177 [============================>.] - ETA: 0s - loss: 0.1060 - n_outputs0_loss: 0.0964 - n_outputs1_loss: 0.0096Epoch 1/100
 44/177 [======>.......................] - ETA: 1s - loss: 0.1357 - n_outputs0_loss: 0.1313 - n_outputs1_loss: 0.0044
Epoch 00014: val_loss did not improve from 0.12570
177/177 [==============================] - 4s 25ms/step - loss: 0.1062 - n_outputs0_loss: 0.0966 - n_outputs1_loss: 0.0096 - val_loss: 0.1357 - val_n_outputs0_loss: 0.1313 - val_n_outputs1_loss: 0.0044
Epoch 15/100
175/177 [============================>.] - ETA: 0s - loss: 0.1013 - n_outputs0_loss: 0.0920 - n_outputs1_loss: 0.0093Epoch 1/100
 41/177 [=====>........................] - ETA: 1s - loss: 0.1331 - n_outputs0_loss: 0.1314 - n_outputs1_loss: 0.0016
Epoch 00015: val_loss did not improve from 0.12570
177/177 [==============================] - 4s 25ms/step - loss: 0.1012 - n_outputs0_loss: 0.0918 - n_outputs1_loss: 0.0093 - val_loss: 0.1316 - val_n_outputs0_loss: 0.1297 - val_n_outputs1_loss: 0.0019
Epoch 16/100
175/177 [============================>.] - ETA: 0s - loss: 0.1000 - n_outputs0_loss: 0.0913 - n_outputs1_loss: 0.0088Epoch 1/100
 44/177 [======>.......................] - ETA: 1s - loss: 0.1233 - n_outputs0_loss: 0.1207 - n_outputs1_loss: 0.0026
Epoch 00016: val_loss improved from 0.12570 to 0.12330, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 26ms/step - loss: 0.1001 - n_outputs0_loss: 0.0914 - n_outputs1_loss: 0.0087 - val_loss: 0.1233 - val_n_outputs0_loss: 0.1207 - val_n_outputs1_loss: 0.0026
Epoch 17/100
174/177 [============================>.] - ETA: 0s - loss: 0.0931 - n_outputs0_loss: 0.0847 - n_outputs1_loss: 0.0084Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.1167 - n_outputs0_loss: 0.1142 - n_outputs1_loss: 0.0025
Epoch 00017: val_loss improved from 0.12330 to 0.11700, saving model to /content/mycar/models/mypilot.h5
177/177 [==============================] - 5s 27ms/step - loss: 0.0932 - n_outputs0_loss: 0.0848 - n_outputs1_loss: 0.0084 - val_loss: 0.1170 - val_n_outputs0_loss: 0.1138 - val_n_outputs1_loss: 0.0032
Epoch 18/100
176/177 [============================>.] - ETA: 0s - loss: 0.0894 - n_outputs0_loss: 0.0811 - n_outputs1_loss: 0.0083Epoch 1/100
 42/177 [======>.......................] - ETA: 1s - loss: 0.1195 - n_outputs0_loss: 0.1167 - n_outputs1_loss: 0.0027
Epoch 00018: val_loss did not improve from 0.11700
177/177 [==============================] - 4s 25ms/step - loss: 0.0894 - n_outputs0_loss: 0.0811 - n_outputs1_loss: 0.0083 - val_loss: 0.1210 - val_n_outputs0_loss: 0.1184 - val_n_outputs1_loss: 0.0026
Epoch 19/100
175/177 [============================>.] - ETA: 0s - loss: 0.0888 - n_outputs0_loss: 0.0805 - n_outputs1_loss: 0.0082Epoch 1/100
 44/177 [======>.......................] - ETA: 1s - loss: 0.1193 - n_outputs0_loss: 0.1180 - n_outputs1_loss: 0.0014
Epoch 00019: val_loss did not improve from 0.11700
177/177 [==============================] - 4s 25ms/step - loss: 0.0889 - n_outputs0_loss: 0.0805 - n_outputs1_loss: 0.0084 - val_loss: 0.1193 - val_n_outputs0_loss: 0.1180 - val_n_outputs1_loss: 0.0014
Epoch 20/100
175/177 [============================>.] - ETA: 0s - loss: 0.0861 - n_outputs0_loss: 0.0780 - n_outputs1_loss: 0.0081Epoch 1/100
 44/177 [======>.......................] - ETA: 1s - loss: 0.1262 - n_outputs0_loss: 0.1225 - n_outputs1_loss: 0.0037
Epoch 00020: val_loss did not improve from 0.11700
177/177 [==============================] - 4s 25ms/step - loss: 0.0862 - n_outputs0_loss: 0.0782 - n_outputs1_loss: 0.0081 - val_loss: 0.1262 - val_n_outputs0_loss: 0.1225 - val_n_outputs1_loss: 0.0037
Epoch 21/100
174/177 [============================>.] - ETA: 0s - loss: 0.0812 - n_outputs0_loss: 0.0735 - n_outputs1_loss: 0.0077Epoch 1/100
 41/177 [=====>........................] - ETA: 1s - loss: 0.1191 - n_outputs0_loss: 0.1174 - n_outputs1_loss: 0.0017
Epoch 00021: val_loss did not improve from 0.11700
177/177 [==============================] - 4s 25ms/step - loss: 0.0817 - n_outputs0_loss: 0.0738 - n_outputs1_loss: 0.0079 - val_loss: 0.1185 - val_n_outputs0_loss: 0.1161 - val_n_outputs1_loss: 0.0024
Epoch 22/100
174/177 [============================>.] - ETA: 0s - loss: 0.0813 - n_outputs0_loss: 0.0734 - n_outputs1_loss: 0.0079Epoch 1/100
 44/177 [======>.......................] - ETA: 1s - loss: 0.1172 - n_outputs0_loss: 0.1146 - n_outputs1_loss: 0.0025
Epoch 00022: val_loss did not improve from 0.11700
177/177 [==============================] - 4s 25ms/step - loss: 0.0810 - n_outputs0_loss: 0.0732 - n_outputs1_loss: 0.0078 - val_loss: 0.1172 - val_n_outputs0_loss: 0.1146 - val_n_outputs1_loss: 0.0025
Epoch 00022: early stopping
Training completed in 0:01:55.


----------- Best Eval Loss :0.116997 ---------
<Figure size 640x480 with 1 Axes>

Step 4 opt B: Train RNN model

The RNN model combines several images to calculate steering and throttle.
Use the manage.py script to start training

!python /content/mycar/manage.py train --type rnn --model /content/mycar/models/mypilot.h5 --aug

Step 5: Check model and transfer data

To check the quality of the model we look at the loss curve and see how well commanded and predicted steering and throttle values match.
We transfer the data to the car and show how to start the self driving car.

Plot loss curve of model

The curve should show smaller loss vs epochs and the train and validation loss should not differ too much.

Tip: If train loss is much smaller than validation loss your model might be overfitting.

%cd /content/mycar/models
file = glob.glob("*.png")
Image(file[0])

/content/mycar/models

Plot commands and predictions

You can use

donkey tubplot <tub_path> [--model=<model_path>]

to plot the commands and predictions of steering and throttle

%cd /content/mycar
!donkey tubplot  --tub=data/ --tub=data/tubVaihingenIIICleaned200126 --model=models/mypilot.h5 
file = glob.glob("/content/mycar/models/mypilot.h5_pred.png")
Image(file[0])

/content/mycar
using donkey v3.1.2 ...
loading config file: ./config.py
myconfig myconfig.py
loading personal config over-rides from myconfig.py

config loaded
"get_model_by_type" model Type is: linear
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.
Instructions for updating:
If using Keras pass *_constraint arguments to layers.
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling Zeros.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling Ones.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
WARNING:tensorflow:From /tensorflow-1.15.2/python3.6/tensorflow_core/python/ops/init_ops.py:97: calling GlorotUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
Instructions for updating:
Call initializer instance with the dtype argument instead of passing it to the constructor
2020-05-28 15:15:21.333668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-28 15:15:21.336208: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.336818: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:04.0
2020-05-28 15:15:21.337135: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-28 15:15:21.339213: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-28 15:15:21.340908: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-28 15:15:21.341246: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-28 15:15:21.352124: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-28 15:15:21.355539: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-28 15:15:21.362433: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-28 15:15:21.362552: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.363208: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.363746: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-05-28 15:15:21.368965: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2200000000 Hz
2020-05-28 15:15:21.369168: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x61aee00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-28 15:15:21.369195: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-05-28 15:15:21.459586: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.460235: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x61aefc0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-05-28 15:15:21.460266: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla P100-PCIE-16GB, Compute Capability 6.0
2020-05-28 15:15:21.460429: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.460998: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
name: Tesla P100-PCIE-16GB major: 6 minor: 0 memoryClockRate(GHz): 1.3285
pciBusID: 0000:00:04.0
2020-05-28 15:15:21.461059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-28 15:15:21.461084: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-28 15:15:21.461104: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-28 15:15:21.461125: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-28 15:15:21.461143: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-28 15:15:21.461159: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-28 15:15:21.461178: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-28 15:15:21.461246: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.462030: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.462513: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
2020-05-28 15:15:21.462570: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-05-28 15:15:21.463567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-28 15:15:21.463592: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
2020-05-28 15:15:21.463603: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
2020-05-28 15:15:21.463726: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.464301: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-28 15:15:21.464825: W tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:39] Overriding allow_growth setting because the TF_FORCE_GPU_ALLOW_GROWTH environment variable is set. Original config value was 0.
2020-05-28 15:15:21.464864: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14858 MB memory) -> physical GPU (device: 0, name: Tesla P100-PCIE-16GB, pci bus id: 0000:00:04.0, compute capability: 6.0)
processing 1000 records:
2020-05-28 15:15:22.373437: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-28 15:15:22.622740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
<Figure size 640x480 with 2 Axes>

Copy the trained model back to Donkey Car (Pi)

Once the training is complete on Colab, download

• mypilot.h5 file from /content/mycar/models/
• myconfig.py file from /content/mycar/

---

files.download('./mypilot.h5')
%cd /content/mycar
files.download('myconfig.py')

Alternatively, you can copy the model back to Google Drive too

!cp /content/mycar/models/mypilot.h5 /content/drive/My\ Drive/myCar/mypilot.h5

Copy the file from your PC or Mac to the Raspberry Pi using Filezilla or scp command.

sftp pi@raspberry.local
cd mycar/models
put mypilot.h5

Start Autopilot on Pi

cd ~/mycar
python manage.py drive --model models/mypilot.h5 --js

Step 6: Bonus - Salient Object Visualization

The salient visualization gives an indication which parts of the image caused the highest activations in the model. Its a good method to understand what triggers the steering and indentify problems

• reflections
• distractions off the track
  Note: It seems like the salient mode doesn’t work for RNN networks

# !pip install git+https://github.com/autorope/keras-vis.git
!pip uninstall keras-vis
!pip install git+https://github.com/sctse999/keras-vis
  
  

%cd /content/mycar
!donkey makemovie --tub data/{tub_name} --model models/mypilot.h5 --type linear --salient

Download the movie to local machine

%cd /content/mycar
!ls -ahl
files.download('tub_movie.mp4')

Or download the file to Google Drive

!cp /content/mycar/tub_movie.mp4 /content/drive/My\ Drive/myCar/tub_movie.mp4