Developer Projects

Off-ice Flag turns a full hockey game into a short review tape. Every time players make body contact, a card appears with four yes/no signs (head or neck involved, slow to rise, left the ice, favoring a limb), the caption that justified them, a one-sentence why, and a decision: Come off pauses the replay, Watch and Play on keep playing. It never names an injury or rates severity; it only reports what is visible, and a trainer makes the call. How it works: the game is split losslessly into parts under the upload cap and uploaded to VAST with our ingest prompt. The VAST DataEngine pipeline segments each part into ~5 s clips, NVIDIA Cosmos captions each clip (steered to describe body contact and the four signs), and YOLO detections and embeddings land in VastDB. Our app polls VAST, sends each caption to Weights & Biases serverless inference for strict JSON signs plus a quote copied from the caption, applies a fixed Python decision rule (the model never picks the word), and asks W&B for a why sentence that Python checks for the quote and against medical wording. Cards appear while the game is still indexing. Trainers can search footage in plain words (VAST hybrid search), ask questions (cached answers, VAST agent ask, or a VAST recap), and jump straight to the clip.

Hazard perception — solution architecture Second Team 41 product on the shared lab App button. Card 1 is Video Recutter (tools/idea-upscaler/SEED.md). This document is card 3. The test plays one indexed clip, records when the tester marks a hazard, then boxes one object from the YOLO sidecar and scores the reaction. It does not generate video and it does not call Cosmos Reason to judge the answer. Purpose A short hazard-perception test over the archive that is already indexed for Team 41. Three modes pick the camera world: Mode What the tester is Clip the search keeps Driving A driver in street traffic capture_type streets, then traffic. Warehouse rows and nyc_bike_gopro-1 are dropped. Forklift driver A driver in a warehouse aisle location warehouse3, camera_id sdg_warehouse_cam-2, capture_type warehouse Warehouse walk A person on foot in that aisle The same warehouse filter The first pass plays the clip and can prime the sidecar. The hazard is chosen when the second pass starts. One hazard is boxed. The score is a local table, 0 to 5, for Slow down, Brake, Wait, Steer away, or Continue. User flow Open the workshop lab page and click App. The launcher is /app/. The third card, Hazard perception, opens /app/hazard/. Code is in tools/hazard-perception/main.py

Cyclists record close passes every day, and almost none get reported: finding the 5 seconds that matter in an hour of footage and filling in a police form takes too long. And nobody tells riders about the risks they take themselves. Witness is designed for the cyclist to check their own safety. It watches their ride footage, finds every risky moment, and works out who caused it, so they can review a report or a lesson before anything leaves the app. The live front page is titled** Cyclist Safety**. The rides loaded here are assumed to be one user's footage: three NYC helmet-cam rides from their cameras. More devices can be registered. Register device stores another camera id on this user's rides (demo only; it stays in the app).

vgrep is a command-line tool for searching video footage with natural-language queries. It indexes local video, then returns matching clips with timestamps and short descriptions. You can also search using an example image or clip. The project combines text and visual search, then checks candidate results before displaying them. It supports filters such as camera and time range, and can show why a result matched. A fully offline demo uses generated synthetic footage and fake backends; real video search needs a configured embedder and verifier. Optional remote backends include CLIP embeddings, YOLO11 object detection, and Cosmos Reason captioning and verification. vgrep searches actions and objects, and does not identify people or faces.

Motivation: safety is important for workers in the construction site. After talking to site supervisors, we understood that the main safety concern is not physical, but in the mental well-being of the workers. The well-being include many things: wage, mood of the day, etc. Site supervisors understand worker wellbeing by either talking to them or from their body languages. While we cannot directly improve the worker well-being, we can at least consistently check their body languages to make sure workers are metally well — hence safe — like their supervisor does.

Somebody wants to ask questions about the items, activities, and scenes appearing in a video. The user uploads a video using an interactive web page (plus chrome extension)  to VAST, which is analysed using an LLM running on Coreweave to create a description of short segments and a timeline of the start of each scene. Clicking on the generated timeline brings up a summary of the items appearing in the scene, plus wikipedia links. This all combines to create an interactive experience for viewers watching videos with amazing overlays.

Behind is a short lesson for cyclists and drivers, taken from footage they could have been in. You press Watch. A New York rider's view, or a San Francisco intersection, plays with nothing drawn on it. When it ends, the street goes black and one light stays on the thing that should change the next move: a person standing in front of a parked mail truck, a truck filling the lane, a rider ahead while a car closes on the right, people in a crosswalk, a car coming through from the right. A voice gives the advice in one line: wait, stay stopped, hold your line, or let it through. A cyclist can replay the moment they would have squeezed a parked truck or swung out in the dark, and hear the move before they need it. A driver can do the same at a crossing: stay put until the stripes are empty, and do not take a gap that is already closing. The clips and detection tracks come from VAST: video in VAST S3, indexed by VAST DataEngine into VastDB. Behind only reads that archive. NVIDIA Cosmos Reason and Cosmos Embed, on CoreWeave GPUs, wrote and embedded the descriptions. YOLO, on those same NVIDIA GPUs, is what the coach watches: the boxes pick the frame, the spotlight, and the sentence. We built Behind in Cursor with SpaceXAI. Weights & Biases serverless inference, from CoreWeave, is there for a written coaching note. The film takes its spoken line from the YOLO tracks, so the voice and the spotlight show the same thing.

Safety Gate watches laparoscopic surgery video in real time — NVIDIA Cosmos spots bleeding and lost view, NVIDIA Nemotron VL (via W&B Inference) tracks the Critical View of Safety checklist, and alerts fire when something goes wrong. Every call is traced in W&B Weave, the case becomes a searchable log with an auto-summary, and each judgement is exported as Cosmos post-training data; it observes and alerts, never advises.

**Omda is a real restaurant. Friday lunch, six cameras, five minutes, 350 clips nobody has time to watch. Ask Omda answers questions about the floor and cites the moments it used: “Did anyone leave without paying?”, “Deleveroo says rider D6 was late, whose fault was it?”, “What would you flag to the owner?” It works by text or voice over any 5-minute, 30-minute or 1-hour window.** **It does three things a human can’t do at lunch rush. Dine-and-dash detection: a party finishes, stands and exits; Omda checks the till camera in between and finds no payment. Delivery dispute resolution: a door-to-counter-to-door timeline for every rider, so the kitchen stops eating refunds it didn’t cause. Owner’s briefing: an auto-written report of bottlenecks, waits and anomalies, each point linked to a clip.** **Every 5-second clip is described by NVIDIA Cosmos Reason, counted by YOLO11 and embedded with Cosmos Embed1, then indexed in VastDB via VAST DataEngine. Our Floor Twin tracker (DGX Spark) gives customers, staff and riders stable IDs, so the agent can stitch one person across rooms and one moment across cameras. Reasoning runs on DeepSeek via W&B Inference; every answer is traced and scored in Weave against ground-truth and adversarial questions, so it cites or says it doesn’t know. Built with Cursor on the VAST workshop skills.**

The problem: London has 25,000 bus drivers, and TfL's goal is that no one is killed on or by a London bus by 2030. Buses already carry cameras, but the footage is usually only pulled after something goes wrong. Nobody has time to watch it all, so the near misses that come before a crash, and the patterns behind them, go unseen. Safety teams often learn about a risky junction or habit only after someone is hurt. SixthSense helps London bus safety team see which buses had a close call and what to do about it.The live map shows every London bus from TfL open data, but only draws the buses that need a look, in order of risk. For each camera clip, SixthSense: - searches **VastDB** for 8 kinds of bus hazard, like a cyclist on the nearside or a pedestrian stepping out - checks with **YOLO11** boxes for annotations, object recognition that road users are really getting closer, and drops false alarms - asks **NVIDIA Cosmos3-Reason** on CoreWeave what happens next, why, and what the driver should do - watches the next clip and scores its own prediction with a model on **W&B** inference, with every step traced in Weave Note: bus positions are live TfL data. The camera clips are the hackathon footage, standing in for bus cameras.

**The problem:** “item missing” claims are one of the costliest headaches in e-commerce, and sellers often lose them by default because they can’t prove what went into the box. **What ProofBox does:** an overhead camera records every mailer as it is packed. The QR label on the parcel is detected and the frame is linked to the order. YOLO tracks each item placed inside, and NVIDIA Cosmos Reason watches the clip to confirm what each item is and that it actually went in. Every verdict, timestamp and clip reference is stored in VastDB on the VAST AI OS. **When a claim lands:** the seller asks in plain text or by voice (“was the charger in order 4821?”) and a W&B-hosted agent returns a per-item verdict, packed or not packed, with the exact clip as evidence, ready to attach to the marketplace or payment dispute. **Built on:** VAST S3 + DataEngine + VastDB for ingest and the evidence store, NVIDIA Cosmos Reason and Cosmos Embed for reasoning and semantic search over packing footage, YOLO on CoreWeave GPUs for detection and tracking, Weights & Biases Inference for the agent, built in Cursor. **Supply chain potential:** the same pattern works at every handoff. Add cameras at pick, pack, carrier handover and returns, and each order gets a video-verified chain of custody. That gives 3PLs and brands proof for mis-pick and shrinkage disputes, carrier liability and insurance claims, plus searchable warehouse footage (“every order where an item was swapped after pick”).

It is a disaster-response app where AI agents analyse drone video of a disaster or war zone, each looking for one thing (trapped survivors, hazards, building damage or blocked roads), and a coordinator ranks what they find so the operator sees the most urgent alerts first, located on a live video feed and a 3D map.

FireScope 3D turns drone footage into an interactive, evidence-linked 3D wildfire analysis platform for emergency responders. Using real footage from the BurnedAreaUAV prescribed-fire dataset in Portugal, our computer vision pipeline identifies burned areas and visible fire activity, projects observations onto real elevation data, and reconstructs fire evolution along an interactive timeline. Users can explore 3D terrain, inspect historical changes, select events and jump directly to supporting video. NVIDIA Cosmos Reason analyzed 41 real video clips to generate scene descriptions and fire-related observations. Cosmos Embed1 enables semantic video search, while YOLO11 detects relevant objects. VAST Data provides video archive search and retrieval through its indexed infrastructure. CoreWeave hosts GPU inference, and Weights & Biases powers grounded AI question-answering. Cursor helped develop and deploy the application on the workshop's Kubernetes infrastructure. The dashboard also features six drone entities and 13 response resources, with cinematic camera views, aircraft movements and operational visualizations. Synthetic telemetry, resource operations, false-color infrared views and experimental projections are explicitly labelled as simulated. FireScope is an exploratory decision-support prototype, not a certified wildfire prediction system.

Drivers circle for parking because nobody knows where the free kerb is. Cities still count it by hand. Kerb Map fixes that with footage that already exists. Connect a dashcam once and its clips sync to VAST storage. VAST DataEngine runs YOLO11, NVIDIA Cosmos Reason and Cosmos Embed1. Our kerb analysis finds parked cars and gaps and applies street rules such as junctions, driveways and crossings. It asks Cosmos only about gaps the rules can’t settle, which means 270x fewer Cosmos calls than captioning every segment. Each stretch of kerb gets a receipt in VastDB: free, occupied, not a space or can’t tell, plus when it was seen, the evidence frame and the decision trail. In the mobile app, “Find parking” shows the nearest free spaces, and drivers confirm or correct them with 👍/👎. Live now: 11,081 receipts in VastDB, from the organisers’ Toronto dashcam drives and street cameras. Checked against people: 120 dashcam labels and 100 street-camera labels. Street camera agreement is 45 of 51 on held-out spots. Built with: VAST AI OS (S3, DataEngine, VastDB), NVIDIA Cosmos and Nemotron, CoreWeave GPUs, W&B Serverless Inference and Weave, and Cursor agents. Who pays: city kerbside teams and parking operators get kerb data without manual surveys. Dashcam owners get cloud backup and a parking finder.** ** **List of tools used:** VAST AI OS, NVIDIA, CoreWeave, W&B and Cursor

**Live map**: Streamers see what’s around them as it happens, including traffic, crowds, accidents and closures. Better calls on the move: They know what’s ahead before they turn the corner, so they can choose where to go and what to avoid. **Fewer surprises**: Streamers can spot trouble early, from unsafe areas to crowds forming to stream snipers closing in. **A digital twin of the city**: The **Mirrorverse** is populated with real cars and people from live data. Anything there is the same as IRL, but we can do anything in Mirrorverse with no consequences. For example, streamers can add 50 people outside a shop, block a road or change the route, then see how it’s likely to play out. THEN they make the decision IRL. **Beyond streaming**: The same tech works for film shoots, live events, red carpets and celebrity appearances, **anywhere crowds and timing matter**.  Sky is the limit :)

Last Seen is a search tool for the first few minutes after someone goes missing inside a building. A member of staff describes the person out loud, and Last Seen searches every camera, shows the people who fit, and asks a quick question when two of them look alike, so the right person gets picked instead of guessed. From there it follows them across cameras on a 3D model of the building. When they walk somewhere no camera covers, it shows how far they could have got, clears the areas the cameras can see are empty, and ranks the cameras and rooms worth checking first, until a camera picks them up again and confirms it is the same person. We built it on **NVIDIA’s **synthetic hospital footage and used only half of the cameras, keeping the rest hidden so we could check our guesses against where people really were. The footage runs through the **VAST DataEngine **pipeline into **VastDB**, which holds the video search index, our tracks and our person index. NVIDIA Cosmos Reason describes each person and checks what it says against two camera views, **Cosmos Embed **matches descriptions to people, **YOLO11 **finds people in each frame, and **Canary **turns speech into the search. **Weights & Biases** serverless inference runs the agent’s reasoning, Weave traces and scores every step, **CoreWeave GPUs** serve the models, and we built the whole thing in **Cursor.**

**Weakness and improvement sport analysis.**** ** We've built a real time video layer on sport footage to highlight weaknesses in game for both teams and to then follow through with the value proposition of how to improve.

NVIDIA VSS watches the same video. Its Long Video Summarization API returns events with start and end times. Replica maps each timestamp to a frame, looks up the camera pose it solved for that frame, and drops a pin in the 3D viewer. Click a pin and the camera flies to where it was filmed. VSS Q&A answers are parsed for times and pinned the same way. Clips are passed to VSS as URLs from a VAST Data bucket behind an https allowlist, so footage never round-trips through the app. The VSS Alert Agent on live RTSP streams, so incidents land on the map in real time.

Scene2Real aims to save lives by turning warehouse fire footage into robot response rehearsals before physical deployment. NVIDIA Cosmos, served through CoreWeave, analyses incident footage and extracts timestamped fire evidence. An operator reviews the findings, confirms a response target and maps it to an authored warehouse zone. The app builds a parameterised MuJoCo scene and validates a constrained robot plan before execution. The robot rehearses the response while the simulator measures aiming error, unintended collisions and duration. Teams can reset and replay the scene, with each result linked to the original footage and explicit scene assumptions.

Cities have countless examples of infrastructure damage: potholes, graffiti, broken signs and so on. In order to figure out what to fix, councils must drive and search for issues manually, costing money and time. England and Wales spend £4.4 billion annually on these kind of issues, with £240 million purely due to finding out late. With the rise of autonomous vehicles, streets are already being recorded at scale. Our product uses NVIDIA Cosmos (running on W&B Inference) to search hours of this data for damage, pinpoint where using the provided GPS data, use Nemotron to score them and decide which crews should be dispatched, then we use NVIDIA cuopt to find the optimal pathway for crews to perform maintenance. Our entire stack is built on Vast infrastructure.

Self-driving cars don't crash on the boring 99% of driving. They crash on the weird 1%: a pedestrian hidden by a turning bus, a horse-drawn carriage, a school bus with its stop sign out. Those moments are buried in hours of normal footage and finding them by hand is slow and expensive. Haystack finds them for you. How it works: 1. Understand: dashcam video is stored on VAST Data (S3 + VastDB). NVIDIA's VSS Blueprint uses Cosmos-Reason2 to describe every 5-second clip and Cosmos-Embed1 to make it searchable. YOLO11 boxes people and vehicles and measures motion. 2. Search: type a plain-English question ("cyclist running a red light") or switch on the Live agent, which scans camera by camera, invents new hunts and flags edge cases as it finds them. 3. Judge: an LLM on Weights & Biases Inference scores each clip for rarity, danger and training value and explains why. Every call is traced in W&B Weave. 4. Human check: one key press (Y/N) confirms or rejects each find. This measures the judge's accuracy, and rejected clips never reach the dataset. 5. Act: export a labelled dataset (clips, boxes, COCO labels, train/val split) logged as a versioned W&B artifact. Built in Cursor, running on CoreWeave GPUs.

Traffic operations centres have hundreds of cameras and a handful of people watching them. Most of what matters, like a car blocking a lane, a pedestrian stepping into traffic, or a lane closed for street work, is never seen. CityLens is a workspace for those operators. It runs standing rules over every street camera: blocked lanes, pedestrian conflicts, cyclists squeezed by traffic, congestion, work zones and lane closures. Search over the VSS index finds candidates, then Cosmos Reason watches each clip again and gives a yes or no with a reason. So far that's 38 confirmed incidents and 106 false alarms that never reach the operator. Confirmed incidents are written to our own VastDB tables and routed to the team that would act, parking enforcement, safety review, signal timing, or a permit inspection. Each camera is judged against its own normal using the YOLO counts, with a storyboard of the whole recording. Click a city to see its cameras alongside live 311 requests and the city's permit and road restriction data. Every work zone the cameras confirm is checked against active permits, so the city can spot unpermitted work. There's also an assistant. It runs on W&B Inference, queries VAST through eight tools, cites clips you can play, and every step is traced in Weave.

Self-driving fleets record petabytes of video, and the rare dangerous moments are what the next model must learn from. Drive Loop finds them, checks them and turns them into training data. How it works: we use the Toronto dashcam footage already stored in the VAST VSS archive (clips in S3 on VAST, detections and search vectors in VastDB). Stored YOLO11 boxes give 558 candidates where a person is in the car's lane, out of 1,080 clips. NVIDIA Cosmos3-Reason watches each closest clip and says where the person is and what the car should do. NVIDIA Nemotron VL, served by W&B Inference, double-checks the frame and removes cyclists and duplicates: 20 events, 8 of 8 correct when checked by eye. Optical flow measures what the driver really did, so each event gets a 0-100 danger score and a driver-versus-model verdict. Engineers review events in a web app: clip with a perception overlay, speed chart synced to the video, and a "who was right?" label that becomes training data. Anyone can write a new detector in plain words, test it on one clip on the shared NVIDIA GPU in about 5 seconds, then run it over the footage. Each event becomes a scenario package for NVIDIA Cosmos Transfer or Omniverse; the simulator studio previews up to 1,080 variants (rain, snow, night, a child). Sending to a simulator is a demo. The app runs on CoreWeave Kubernetes, judge calls are traced in W&B Weave, and we built on the event's Cursor agent skills.