Hackathon team · StarkHacks 2026 · Apr 2026
ControVirtual
XR teleoperation and a fine-tuned SmolVLA policy for an SO-101 robot arm, built at StarkHacks 2026.
Team project. I worked on the robotics integration and the SmolVLA fine-tuning and inference code.
- Unity
- Meta XR
- Python
- OpenCV
- LeRobot
- SmolVLA
- WebSockets
System
- Quest voice
- WebSocket bridge
- Camera + twin
- SmolVLA
- SO-101 arm
Problem
Our team wanted one setup where you speak a command in a Meta Quest headset, watch the arm in a browser-based digital twin, and run a vision-language-action model on a real SO-101 arm. We had a weekend, so we fine-tuned a pretrained model, SmolVLA, on one task instead of training a policy from scratch.
What I built
- A Unity Quest client that captures speech with the Meta Voice SDK and Wit.ai and sends JSON commands to a local Python WebSocket bridge.
- A Flask and OpenCV server for the camera feed and robot state, and an A-Frame digital twin in the browser.
- Fine-tuning SmolVLA on 50 teleoperated episodes of one task: pick up a screwdriver and place it on a sheet of paper. Task success went from 0% to about 75%.
- Simulated telemetry so the Quest app and the twin can be demoed without the arm connected.
Results
- teleoperated episodes
- 50
- task success after fine-tuning
- 0 → 75%
- Best Use of AMD Tech, honorable mention
- AMD
Limits
The fine-tuned policy covers one task. The full chain from a spoken command in the headset to policy-driven arm motion was still experimental when the hackathon ended.