This lab note is meant to be a brief documentation on my experiments with TFLite and MediaPipe. I'll go over the following:
A couple of years ago, I heavily relied on AWS EC2 Spot instances (g2.2xlarge) with EBS storage for the training assets, and the model weights. Unfortunately the cost of training was quickly going up, and back then Google Colab wasn't quite a thing (nor any of Google's TPU offerings).
After running some continuous experiments on the cloud, I reached the point where building my own machine would be more cost effective – so I got myself a GTX 1080 Ti, and built my own ML rig.
I ssh into my machine, then use tmux to set-up some persistent sessions. Within a tmux session, I launch my jupyter server with jupyter notebook --no-browser --port=8888 --ip=0.0.0.0. The --ip is set to 0.0.0.0 so I can check on my notebooks while on the go.
An alternative to using the --ip option stated above, is to use ssh port forwarding: ssh -NL 8888:localhost:8888 username@64.233.160.0, which will map your local 8888 port to the server's 8888 port. You can then access https://localhost:8888 on from your host.
MediaPipe is a graph based cross platform framework, that facilitates building a pipeline for your ML apps. I was sold on the graph based concept, and overall seemed like an interesting tool to experiment with.
Note: You can use TFLite without MediaPipe but there's more to just adding a TFLite model to building an ML based mobile app.
While the concept is great, MediaPipe has quite the steep learning curve. You need to understand Bazel, write C++, learn the MediaPipe specific concepts such as Calculators etc.
I spent quite some time trying to set-up my project. I initially wanted to treat it like a third-party dependency, but I quickly ran into problems when trying to write my own custom Calculators, or extend their FrameProcessor.java file.
In the end, I set-up my project to have two remote end-points. origin points to my github, while upstream points to google/mediapipe, and I occasionally run git rebase upstream/master to rebase my repo atop MediaPipe.
Note: To reference MediaPipe from within your bazel build files see here.
This following one-liner compiles all your calculator, bundles your .tflite model, streams the final .apk onto your phone, and launches the MainActivity.
bazel build -c opt --config=android_arm64 //mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:helloworld --sandbox_debug && adb install bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic/helloworld.apk && adb shell am start -n com.google.mediapipe.apps.basic/com.google.mediapipe.apps.basic.MainActivity && sleep 2 && adb logcat --pid=`adb shell pidof -s com.google.mediapipe.apps.basic`
I've also gone ahead and created a new calculator that converts std::vector <GlBuffer> into mediapipe::GpuBuffer. This is great for models that spit out an image. I learnt about Compute Shaders – something that hasn't quite made its way to WebGL!