camera, code, action
backstory: I was a Matlab-on-Windows machine vision developer and I wanted to code Python and Bash on hardware, then came the cloud and TensorFlow.
At the end of this post, there is a link to download training data collected by
blue1. There is also a link to download a model that is trained on this data and has run on
build a low-cost rover that carries a camera, runs deep learning vision models through python and bash, and is cloud-connected.
blue1 runs a modified version of this
asyncio loop that captures frames that it feeds into a 3-class TensorFlow classifier named
blue_driver to receive one of the following labels:
turn_left, turn_right, no_action, which are then communicated w/ the motor controllers through a serial line.
blue1 is operated through a standard keyboard. User presses
w to start the motors and then steers the robot by pressing
a (to turn 10 deg left/right). The frames captured by the camera are annotated by these keystrokes and a cloud worker runs a nightly retrain of
blue_driver triggered by the availability of additional training data. The latest
blue_driver is then released through an MLFlow-style model lifecycle-management backend. The same backend enables
blue1 to always pull the latest
At runtime, the user can issue a prediction request on the current frame by pressing
m. The results of this prediction will be implemented in realtime, i.e.
blue1 will turn left or right or will continue its path based on the output of
blue_driver on the latest frame. There is also a continuous prediction mode,
n, which runs at
~3fps. Here is how
blue1 looks like when it is driving autonomously:
Parts of the terraform of
blue1 + the design of
blue_driver are courtesy of Donkey Car.
As I did more research on the electronics parts and the software and cloud concepts and components relevant to building
blue1, I started to see a bigger picture:
a minimal python+bash machine vision back+frontend that terraforms common Linux machines and enables edge data collection and model execution.
more on this in future posts.