Tandem: AI Assistance in Physical Space
2026
Python · Gemini Live · Multimodal Knowledge Retrieval · Spatial Tools
Connecting multimodal AI, domain knowledge and spatial tools to help technicians work with the equipment in front of them.
- My role
- AI engineering and multimodal knowledge retrieval
- Context
- Company project
- Company / Studio
- Trifork
- Status
- Ongoing development
Expertise where the work happens
Tandem by Trifork connects onsite technicians with remote experts from their team through XR devices. An expert can see what the technician sees and guide them across locations and borders — particularly valuable when a small group of specialists supports sites around the world.
The AI assistant offers another way to access that expertise, combining the technician’s visual and spoken context with the company’s domain knowledge. Multimodal knowledge retrieval connects the situation in front of the technician with relevant information.
A key part of the approach is making knowledge from remote expert sessions reusable. This turns individual expertise into shared knowledge that the assistant can draw on when similar problems arise.
Building the AI layer
As one of the primary engineers working on Tandem’s AI assistant, I helped bring multimodal models, knowledge retrieval and spatial interaction together with the team. I worked on both the on-device assistant for Apple Vision Pro and the device-agnostic, Python-based Gemini Live backend, integrating domain knowledge and visual context into live conversations.
This involved model integration, multimodal knowledge retrieval, real-time audio, interruptions and session handling, balancing responsiveness, quality and precision.
Guidance alongside the task
The assistant is designed to identify relevant components, point to them in the technician’s surroundings and bring supporting documents, blueprints and schematics into view alongside the task. This keeps guidance close at hand while leaving the technician’s hands free.