2025/04/09 Meeting Notes

[himanshunaidu]

Progress Update

1. Computer Vision ML Pipeline
   Continued further work on incorporating training and testing logic in the ML pipeline. Also incorporated multiple evaluation metrics such as mIoU, mDice, and ROM/RUM.
   The current code can train a model similar to that of the EdgeNets code on the CityScapes dataset.
   Still need to incorporate custom datasets into the training as well as testing pipelines.

The model conversion from PyTorch to CoreML is finally working with good performance for models such as BiSeNetv2.
Now we have a BiSeNetv2 CoreML model ready for usage as a test version.

2. Segmentation Post-Processing
   Have analyzed segment_streets.py and other similar files in edge_mapping repository
   Started working on implementation of object (centroid tracking) on iOS using OpenCV iOS.

The post-processing logic may prove to be another bottleneck in real-time processing.
E.g. Watershed Algorithm (instance segmentation): 20 ms per frame

3. Other Changes
   Found a computation bottleneck in the real-time segmentation post-processing.
   Fixed it with iOS Accelerate.
   Real-time Post-processing of each frame for the current logic has been reduced from 50 ms to < 5 ms.

Meeting Notes

1. Computer Vision ML Pipeline
   In order to incorporate vision transformers in the pipeline, we should explore possible options.
   MLX seems to be a good option to explore. It is open-source.

Another option for converting PyTorch models to edge device-friendly format is Executorch.
https://pytorch.org/executorch/stable/demo-apps-ios.html
(Not sure if it will help deploy transformers, but it may be worth a try)

2. Segmentation Post-Processing
   OpenCV is a little too outdated in general, plus it seems to be moving to a payment model.
   Moving most of the logic to Metal shading language may be possible.
   E.g. To get contours of an image, convert the image to black and white, shrink the image by 1 pixel on all corners to get a reduced image, and subtract the reduced image from the original image to get the contours.

Next Steps

1. Explore Executorch

2. Explore Metal Shading Language
   https://developer.apple.com/documentation/metalperformanceshaders
   https://developer.apple.com/metal/