This lab note is meant to be a brief documentation on my experiments with TFLite and MediaPipe. I'll go over the following:
A couple of years ago, I heavily relied on AWS EC2 Spot instances (
g2.2xlarge) with EBS storage for the training assets, and the model weights. Unfortunately the cost of training was quickly going up, and back then Google Colab wasn't quite a thing (nor any of Google's TPU offerings).
After running some continuous experiments on the cloud, I reached the point where building my own machine would be more cost effective – so I got myself a GTX 1080 Ti, and built my own ML rig.
ssh into my machine, then use
tmux to set-up some persistent sessions. Within a
tmux session, I launch my jupyter server with
jupyter notebook --no-browser --port=8888 --ip=0.0.0.0. The
--ip is set to
0.0.0.0 so I can check on my notebooks while on the go.
An alternative to using the
--ip option stated above, is to use ssh port forwarding:
ssh -NL 8888:localhost:8888 firstname.lastname@example.org, which will map your local
8888 port to the server's
8888 port. You can then access
https://localhost:8888 on from your host.
MediaPipe is a graph based cross platform framework, that facilitates building a pipeline for your ML apps. I was sold on the graph based concept, and overall seemed like an interesting tool to experiment with.
Note: You can use TFLite without MediaPipe but there's more to just adding a TFLite model to building an ML based mobile app.
While the concept is great, MediaPipe has quite the steep learning curve. You need to understand Bazel, write C++, learn the MediaPipe specific concepts such as Calculators etc.
I spent quite some time trying to set-up my project. I initially wanted to treat it like a third-party dependency, but I quickly ran into problems when trying to write my own custom Calculators, or extend their
In the end, I set-up my project to have two remote end-points.
origin points to my github, while
upstream points to
google/mediapipe, and I occasionally run
git rebase upstream/master to rebase my repo atop MediaPipe.
Note: To reference MediaPipe from within your bazel build files see here.
This following one-liner compiles all your calculator, bundles your
.tflite model, streams the final
.apk onto your phone, and launches the
bazel build -c opt --config=android_arm64 //mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:helloworld --sandbox_debug && adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic/helloworld.apk && adb shell am start -n com.google.mediapipe.apps.basic/com.google.mediapipe.apps.basic.MainActivity && sleep 2 && adb logcat --pid=`adb shell pidof -s com.google.mediapipe.apps.basic`
I've also gone ahead and created a new calculator that converts
std::vector <GlBuffer> into
mediapipe::GpuBuffer. This is great for models that spit out an image. I learnt about Compute Shaders – something that hasn't quite made its way to WebGL!