Developer Projects

Time Search lets you search camera recordings in plain English on top of VAST and Coreweave architecture. Describe someone's clothing, review likely matches, and follow their sightings across cameras on a shared timeline. Every result links to the original clip and timestamp, so you can check what happened without scrubbing through long recordings.

Put Me In The World connects a personalised 3D avatar with real street-video evidence. A browser-based pose estimate can personalise illustrative body proportions. VAST retrieves timestamped street clips; existing Cosmos descriptions and YOLO detections provide scene evidence. Caption keywords suggest terrain and obstacles, while a live Cosmos call recommends the next scenario and a button runs the next search. Users explore five device concepts, including a hands-free bypass aid, with authored movement previews and transparent rule-based rankings. A connection panel explains how street constraints relate to movement and device review questions. Built with Claude Code and Cursor, Three.js, MediaPipe and a Flask backend on the Team 6 stack. Limits: the avatar is not a body scan; video is not reconstructed or registered in 3D; movements and rankings are illustrative, not validated biomechanics or clinical recommendations. W&B runtime logging is not claimed.

Collecting humanoid data is hard and expensive. We created real-time data collection pipeline, it corrects the collector if they make a mistake in real time (visual "unit tests"). A voice guide walks collectors through a task, vision checks each placement We use LLMs and Vision LLMs hosted on Weights&Biases as part of this. All videos are then uploaded via API to VAST platform for navigation, exploration and quality assurance (see https://real-sim-physical.vercel.app/library which uses VAST platform). The videos are then used to train NVidia GR00T model and we demonstrate that GR-1 model works with trained model in NVidia's simulator. The model runs on GPUs on Modal.com.

**What we built** LiveGuard checks, during a live video call, whether the person on screen is a real human, before a finance approver releases a payment. It runs as a small floating dot over Meet or Zoom. The caller is never asked to do anything. The agent watches, decides, stays quiet on a pass, and leaves a red banner up on a fail. **How it works** **Watch. **The browser records the caller's video tile, from a screen share of the call or a webcam, in back-to-back 8-second windows. **Measure,** with YOLO. YOLO pose finds the caller (the largest person in the window, ignoring your own self-view) and checks four things: 1.Is there one clear face? 2.Are there no sudden jumps between frames (cuts or glitches)? 3.Does the head show natural micro-movement? A held-up photo moves as one rigid piece. 4.Are the frames changing at all, or is the feed frozen? It also marks the hard moments, a head turn or a hand near the face, because that's where face-swap deepfakes break. Judge, with Cosmos. NVIDIA Cosmos sees five frames, hard moments first, each labelled with what YOLO measured. It judges whether the person looks physically real, and names any artefact with its frame number.**Decide each window.** 1.FAIL if Cosmos found an artefact or the feed is frozen. 2.INCONCLUSIVE if there was no clear face or no natural movement. 3.Otherwise PASS: "strong" if a hard moment was judged clean, "basic" if the caller stayed still.

CURRENT watches a real Arduino Nano build and stops you before you power a bad circuit. A camera looks down at the breadboard. NVIDIA YOLO11, on CoreWeave, wakes the pipeline only when the board changes. NVIDIA Cosmos3-Reason reads the frame into a circuit twin: parts, pins, and wires. A deterministic checker (circuit rules and a small solver) decides the fault, and it speaks only after two looks agree, so a flicker is not a false alarm. Weights & Biases phrases that verified finding and does not invent one. The 3D twin highlights the next pin. Flip the LED or pull the 220 ohm resistor and CURRENT says so out loud, marks the bad wire, and shows it moving to the right place: D13, resistor, LED anode, cathode to GND. Hold the mic and ask anything. NVIDIA Canary transcribes the question, and the coach answers from the live board. Ask why the resistor is there, or what changed since the last step. VAST Data keeps the whole session. Twin snapshots and key frames go to VAST S3, and past builds are searchable. NVIDIA Cosmos-Embed recalls a mistake you already made, so a repeated backwards LED is caught faster. A Blink simulator checks the same contract off-camera: missing resistor, wrong pin, open ground. A diagnostic mode searches the VAST index and says plainly when a clip is not a circuit. Sabotage the board. CURRENT should interrupt before you plug it in.

Two unsupervised detectors scan traffic videos for the few seconds that matter

ReStrike is an AI-powered tennis analysis tool that doesn't just show you what happened. It helps you explore what could have happened.  Here, we're using VAST to process tennis match footage and identify potential net errors. Now, instead of just watching the mistake, we can explore different possibilities. What if the player hit the ball a little higher? What if they made contact earlier? Or changed the speed? Our simulator visualizes how these changes could affect the ball's trajectory—and whether it might clear the net.

FireFront 3D brings wildfire footage, camera views, and investigation evidence into one workspace. Its Python agent connects questions to timestamped evidence; the current demo uses fixtures while live integration remains pending. We use Cursor to develop the application. VAST Data provides incident video storage, metadata management, and retrieval. NVIDIA Cosmos provides video descriptions and embeddings through the existing VAST pipeline

ET Phone Desk turns screen recordings into resolved incidents. When a trader hits a hotkey, the last 60 seconds of their screen is ingested into the VAST AI OS, which indexes every clip for semantic search, so a growing archive of past incidents and their fixes becomes searchable in natural language. NVIDIA Cosmos reasons over the clip to identify the application, the steps taken, the error code and the exact moment of failure, with OCR picking up on-screen text. Weights & Biases–hosted LLMs then decide what to do: deterministic rules map known error codes to a team, a priority and a safe fix playbook, the LLM handles only ambiguous cases, and an independent verifier agent double-checks every decision. For eligible issues, the agent runs an allowlisted, reversible fix and confirms it worked with a health check before marking it resolved. Critical issues are routed to the owning team in Slack immediately, with the clip, the diagnosis and everything already tried. Guardrails keep it safe on a trading desk: on-screen text is treated as untrusted, the agent never touches orders or positions, and per-clip cost and time budgets apply. W&B Weave traces every step and scores routing and resolution accuracy against our labelled clips, all running on CoreWeave GPUs and built with Cursor and Claude Code.

Sherlocked transforms warehouse surveillance footage into actionable safety intelligence. Instead of manually reviewing hours of recordings, safety teams can monitor multiple camera feeds through an interactive command centre that flags potential collisions, near misses, and hazardous situations. When an incident is flagged, Sherlocked highlights the relevant footage, identifies involved objects, and helps investigate possible contributing factors. It then recommends preventive measures, supports risk assessments, and helps create corrective-action plans. Built on **VAST AI OS**, Sherlocked leverages **NVIDIA Cosmos** video understanding, **YOLO** object-detection metadata, and **CoreWeave** infrastructure. **Weights & Biases** powers AI-assisted incident analysis and safety recommendations, while **Cursor** accelerates development. Our goal is to go beyond video search: **detect the risk, investigate the incident, and help prevent the next accident.**

Developing Hazard puts an AI through the hazard perception part of the UK driving theory test. It streams PIE dashcam clips from the VAST index and clicks when it spots a developing hazard. Each click is scored with the official DVSA rules: up to 5 points per hazard, more for spotting it earlier, and zero for clicking non-stop.

**What we built:** GeoTrace works out where a dashcam clip was recorded from the video alone, with no GPS and no metadata. It reconstructs six 15-minute Toronto trips on a map, to a median of 10 m when we hide a sign and predict where the car was. It can also check a claim, such as “was this recorded at X?”: a made-up location came back 557 km off. We also built TripScore, which gives each trip an explainable driving-risk score from the same footage.**How it works:** Cosmos3-Reason reads street signs, business names, tram tracks, motion and turns from every 5-second segment. An anchor-chain solver then matches the sign reads to the OpenStreetMap road network and rejects misreads that don’t fit the distance and timing. Optical flow measures turns independently, and a second model re-checks each sign from the clip’s frames. An independent business-name check agrees for 38 of 73 names within 300 m. The app is deployed on the team cluster.**Sponsor tech:****NVIDIA:** Cosmos3-Reason powers the sign and motion reading, plus the fault review in TripScore. YOLO11 detections drive TripScore’s tracking, and Nemotron-3.5-Super-VL-120B is the second opinion.**CoreWeave and Weights & Biases:** the Cosmos and YOLO endpoints run on the CoreWeave GPUs, and we called Nemotron through W&B Serverless Inference.**VAST Data:** we built on the pre-ingested index, S3 segments and detection sidecars through the VSS API, and deployed to the team cluster with the challenge’s skill.

Cysense turns cyclists’ cameras into shared road intelligence. As you ride, NVIDIA Cosmos watches your phone’s camera and Clover, your ride companion, speaks up about hazards: blocked bike lanes, roadworks, car doors and vehicles cutting close. Every report is pinned to the map with the video evidence behind it, and each one is tracked over time as it’s confirmed, goes stale or clears. Riders coming up behind get a heads-up or an automatic reroute before they reach it. In the recorded NYC replays, a hazard filmed by one bike changes the next rider’s route. In live mode, you search for a destination, Cysense draws a bike route with turn-by-turn directions, and Clover warns you and reroutes around blocked roads as you go. The stack is NVIDIA Cosmos for video understanding, VAST for the searchable footage archive, and W&B Weave tracing every model call.

My Day is a dignity-first personal episodic memory agent designed for older adults living independently or experiencing early memory changes. The application transforms short daily video clips into a searchable timeline, allowing users to ask natural language questions—such as "Has Sarah visited today?" or "Where are my keys?"—and receive calm, grounded answers backed by direct video clip proof. Built on a tech stack featuring VAST AI OS vector search, NVIDIA Cosmos video understanding, CoreWeave GPU compute, and Weights & Biases agent logic, the system strictly prevents hallucinated memories by safely abstaining whenever an event wasn't filmed. Crucially, all video logs remain 100% private until the user explicitly chooses to share specific clips with family carers, replacing intrusive 24/7 room surveillance with dignity, individual control, and peace of mind.

Problem Warehouse CCTV is long. Mixed traffic (people + forklifts/AMRs) is dangerous in an instant. Operators should not scrub full takes. What we built Shared Zone Watch sits on the team-44 VAST Video Search archive. We pin three ceiling/eye cameras from the lab corpus and open only the trigger window: • ceiling_04 — 0:05–0:10 — person steps into the robot path (NEAR MISS) • ceiling_01 — 0:05–0:10 — human and vehicle in the same corridor (SHARED ZONE / yield) • eye_00 — 0:00–0:05 — contact already happened (COLLISION) Play the 5 seconds. Close the rest of the video. Extra search hits for “person too close to a forklift or robot” can appear below the pinned cameras. How it works VAST + NVIDIA Cosmos/YOLO already indexed the footage at ingest. We do not re-run models on click. The app authenticates to the team VSS API, resolves those clips, proxies playback, and seeks to the known timestamps. The product is the operational call: yield, stop, or review — not another generic search box. Stack VAST AI OS (index, search, stream) · NVIDIA Cosmos / YOLO in the existing pipeline · Cursor on the lab VM · app deployed on the team Kubernetes Ingress at /app (workshop App button). Weights & Biases credentials were available in the environment for agent logic. Demo line “Don’t watch the aisle. Watch the second it goes wrong.”

**Relive: Your camera roll remembers everything. Now it can tell a story.** Meet your camera roll’s slightly dramatic sidekick. You’ve got 14,000 photos and videos sitting in cloud purgatory. That trip to Tokyo. The birthday cake that became a fire drill. Forty minutes of cycling footage in which you were absolutely convinced something interesting happened. Somewhere in there is a great story. Unfortunately, it’s surrounded by pavement. **Daystory turns “I should do something with these videos” into something worth watching.** Start with an idea: “Make my bike ride feel like a nature documentary about a creature desperately trying to arrive at work on time.” Daystory looks through the available footage, finds moments that support your idea, and drafts a storyboard with a beginning, a running joke, and a payoff. You review the scenes, change a line, swap a clip, or tell the narrator to take itself slightly less seriously. Then the edit takes shape. The uneventful stretch of road speeds past. A motorcycle enters the frame, and the film settles into normal speed. A tiny roadside detail gets a little breathing room—and a completely unnecessary amount of dramatic significance. “Here we observe the commuter pursuing an increasingly endangered species: punctuality.” “The motorcycle has already been promoted. Our subject remains under review.” “Even the fire hydrant has a permanent position.” **The footage supplies the facts. The narrator supplies the attitude.**

What We BuiltMicroDuck uses real-world animal videos to guide a simulated robot’s responses, combining video search, playback, agent-driven actions, and physics simulation. We ingested three Hugging Face datasets totaling **X minutes of video**. Why It MattersRobot training needs diverse interaction data. MicroDuck explores using animal videos as behavioral references for embodied AI. Potential applications include long-term, close-range wildlife observation and pet companionship. How It Works**Encounter Mode:** Retrieve an interaction video, use one animal’s behavior as the situation and the other’s as a response reference, then execute MicroDuck’s closest supported action.**Loop Mode:** Turn each completed action into the next search prompt and repeat.The agent harness coordinates retrieval, interpretation, action selection, and execution feedback, using existing pretrained movement policies to explore future training workflows. Technology**Simulation:** Pollen Robotics’ MicroDuck simulator, MuJoCo WebAssembly physics, Three.js rendering, and ONNX Runtime Web movement policies.**Agent and evaluation:** Codex with MCP search tools; Weights & Biases for tracing and evaluation.**Video infrastructure:** VSS, VastDB, and S3 for searchable clips, captions, metadata, detections, and playback; VAST DataEngine with Cosmos3-Reason, YOLO11, and Cosmos Embed1 for descriptions, detections, and embeddings.**Application:** React and TypeScript frontend with a Python FastAPI backend.

🦮 Guidog — Your AI Guide, Right in Your Pocket👁️ The ProblemOver **2 million people in the UK** live with sight loss. Navigating busy streets can be challenging, while guide dogs cost approximately **£102,000** over their working lifetime. Existing AI wearables can be expensive and inaccessible. **What if greater independence was already in your pocket?** 💡 Our Solution**Guidog turns your smartphone into an affordable AI mobility companion.** Using real-time computer vision, Guidog: 👁️ **Sees ahead:** Detects potential obstacles and hazards. 🔊 **Speaks up:** Delivers simple audio cues like "Stop!" or "Wait!" 📱 **Costs less:** Works with everyday smartphone hardware. ⚙️ How It Works**📱 Camera → 🧠 AI Vision → 🔊 Audio Guidance → 🚶 Safer Decisions** 🌍 Our VisionGuidog complements, rather than replaces, guide dogs and white canes. Our mission is to make AI-powered mobility assistance accessible to everyone. **🦮 Guidog — More awareness. More confidence. More independence.**

live agent helps visually repaired

**Grokham brings video search and live street intelligence into one interactive console.** ** ****Query indexed footage in plain language, play matching clips, and ask grounded follow-up questions. Detect objects with YOLO11s and follow partial vehicle tracks with 3D map overlays.** ** ****LIVE from NY streets: explore NYC camera feeds on an interactive 3D Google map, with linked camera selection, video, detections, and chat.** ** ****Our creative feature is inter-camera VLM agent communication: one camera reports what it sees; its two nearest neighbors reply from their own video summaries. Compact yellow speech bubbles and highlighted cameras make the exchange visible on the map.** ** ****Built on the challenge stack: VAST Data / NVIDIA VSS for indexed video retrieval, NVIDIA Cosmos for visual reasoning, CoreWeave-hosted GPU inference, and Cursor for development. A unified UI connects query, detection, partial tracking, maps, and camera-agent dialogue.**

RecallAI is a memory layer for smart AI. A phone camera, or glasses, watches wearer's day, keeps only moments that matter, and turns them into structured memory. Capture and filter -> perceive using VLM -> remember using object events -> reason using agent -> prove answers using frames relied on. NVIDIA used for perception step with Nemotron VLM core of system. Cursor used to develop in IDE. CoreWeave and Weights and Biases calls run on serverless inference, and powered by GPUs.