Developer Projects

**Inspiration:** Companies record how their best people work, and almost nobody watches it. New hires still learn by shadowing, and mistakes get caught late. We wanted that unwatched video to coach the next person live. How each sponsor is used: - **VAST Data (VAST AI OS):** stores the 2,352-clip archive and our own recorded takes in S3. We segmented, captioned, embedded and inserted our takes into the team VastDB ourselves (2,409 rows). VAST semantic search powers "Ask the archive". - **NVIDIA Cosmos Reason:** live scene reading about once a second during coaching, plus clip captions and task recognition. - **NVIDIA Cosmos Embed:** vectors for the archive map, task recognition and search. - **CoreWeave:** GPUs serving all the Cosmos and YOLO models. - **Weights & Biases:** serverless inference. Llama 3.3 70B writes the score feedback and names the map clusters, DeepSeek V4 Flash is the fallback judge, and Weave traces every live check. - **Cursor / SpaceXAI:** Used for creating the project

The problem. Warehouse workers are hurt at twice the national rate (4.6 vs 2.3 injuries per 100 workers, BLS 2024), and forklifts killed 84 workers in 2024. Today a hazard is usually found by an injury, an OSHA complaint or an inspection, and federal OSHA would need 191 years to inspect every workplace once. What we built. Eyes on the Floor is a video agent that reads the cameras a warehouse already has and flags OSHA hazards as they happen. Our demo detects two: a blocked emergency exit (29 CFR 1910.37) and a person on foot in a forklift lane (29 CFR 1910.178). How it works. Frames from [video source] go to [model], which [how it detects, e.g. tracks people and forklifts and checks them against marked zones]. When a hazard appears, the agent names the OSHA rule, the camera and the location, and alerts the floor lead through [alert channel]. Every flag is logged with a time stamp. Technologies. We used [technology 1] for [what it does], [technology 2] for [what it does] and [technology 3] for [what it does]. Why it matters. The floor lead hears about a hazard in seconds, while it can still be fixed, and before anyone has to file a complaint.

CityPulse turns urban video archives into an interactive visual storytelling agent. A user asks a question such as “How do pedestrians and vehicles interact at Toronto intersections?” CityPulse searches the indexed Toronto driving footage, surfaces the strongest evidence, selects six complementary shots, writes a grounded narration, and automatically edits them into a 30-second narrated film. The interface makes the agent’s work visible: source routes illuminate as relevant moments are found, evidence clips appear during retrieval, selected shots are ordered into the final cut, and the resulting film answers the original question. CityPulse builds directly on the VAST Video Search & Summary stack. VAST DataEngine stores and retrieves the indexed video evidence. NVIDIA Cosmos Embed powers semantic video retrieval, while Cosmos Reason provides scene understanding and descriptions. YOLO11 detections provide object-level evidence for vehicles, pedestrians, cyclists and other road users. Weights & Biases serverless inference on CoreWeave powers the higher-level AI director that selects shots and writes the narration. The final response is not generated footage: every shot comes from the licensed source archive and remains grounded in retrieved evidence.

One laptop camera watches the class. On the laptop, YOLO11-pose, YOLOE and YAMNet track the student and spot loud things by name (a blender, a drill) before they make a sound, plus crowding, someone rushing in or someone staying too close. As risk rises, calming sound fades into the student's AirPods early and the teacher's phone gets an alert with the right IEP accommodation. Covering ears or rocking triggers an overload alert. NVIDIA Cosmos3-Reason on CoreWeave GPUs reads the last few seconds of video and says what is about to get overwhelming. Canary-1B transcribes the lecture, so a student with ADHD can ask "what did I miss?" about the minutes they looked away; Llama 3.3 70B on W&B Inference answers. Weave traces every model call. On VAST, we scored all 2,352 clips in our archive for sensory load from the Cosmos captions and YOLO11 counts (771 high, 566 calm) and built a sensory map: the SF street cameras are the most overwhelming (76% high load), the neighborhood camera the calmest (97% calm). Our VM app adds a Cosmos risk timeline over archive video, VAST search with clip streaming, re-ingest with our sensory prompt, and W&B summaries of the calmest places and times.

Riders rate from the back seat and can't see why a driver stopped; telematics knows the car braked, not that a pedestrian stepped out. Dashi reviews each trip's dashcam footage and finds the moments that matter. Proactive side: kudos cards and lesson cards built from the driver's own clips, posted without being asked. Ask side: the driver says "Help me improve my rating" and Dashi finds the 3-star trip, plays the clip where they yielded to a pedestrian, and offers to send the evidence for a rating review. Footage questions in plain English ("show me pedestrians crossing in front of me") run a live VAST hybrid search. For a coachable moment (turning while a pedestrian was still crossing) Dashi shows a better way from a fixed playbook and writes an Ideal replay prompt for a Cosmos Predict/Transfer-style model to render the same scene driven safely. Every verdict quotes the camera caption word for word. Eval on 12 labelled clips: verdict 12/12, scenario 10/12, 0 false alarms; groundedness of summaries 3/7 (our weak spot). Footage is the pie_cam-3 Toronto research dashcam; ratings, routes and driver are simulated. No telemetry, so no speed or braking claims.

**FreezeFrame turns scattered CCTV recordings into a synchronized spatial replay, where investigators can inspect a disputed moment and trace every finding back to the original footage. ** It often happens that insurance companies and victims get deadlocked because there is not enough high quality evidence proving an incident or disproving an incident. It takes a lot of time and energy on everybody's behalf to validate whether a particular claim is accurate or not. Usually whatever information that they have from CCTV cameras can be disputed by lawyers and in the end nothing really happens. As the evidence required  here is accurate, undeniable evidence of an incident happening or not happening. FreezeFrame uses vast data's infrastructure to look up specific incidences. Find all similar incidence videos from several CCTV cameras. Go through them, reason about the exact frames that represent an incident. And then actually pick the best frames and create a Gaussian splat, which is a 3D reconstruction. So the incident can be looked at from all angles, and it is indisputable evidence. This will lead to faster turnaround time for insurance claim cases and hence helping a lot of people and companies in the process. Project uses VAST data's video vector search, nVIDIA's Cosmos Reasoning to get temporal information about what's happening in the video, the is built using Cursor.

Hop24 turns **real highway footage into an arcade game**. Players control an animated chicken crossing traffic recorded on Nashville’s I-24 corridor, timing hops between vehicles. The unusual part is that the cars and trucks are not manually created obstacles: before gameplay, **YOLO/FastSam** detects vehicles in each video frame, recording bounding boxes, labels, confidence scores, and timestamps. The browser combines video playback with a canvas showing the chicken and game elements. Saved detections drive collision detection, while autopilot uses recent vehicle positions to estimate movement and decide when the chicken should advance, wait, or retreat. A collision starts a second experience: investigation. Hop24 replays the event, captures evidence, uses an AI-generated video description to build a query such as “white truck,” and searches an archive for related footage from other cameras. Results can be collected into an incident report. The system uses **VAST **for storage and processing**, VastDB **for detections and searchable data, **NVIDIA Cosmos **Reason for video descriptions, **Cosmos Embed1** for representations, and CoreWeave GPUs. Most AI processing happens before gameplay, keeping the interactive experience responsive. Hop24 demonstrates how detected objects can connect gameplay, video search, and investigation—while making clear that similar vehicles are only candidate sightings and that the chicken collision is fictional.

Give your legacy app a second life—with a video-guided agent. Invoice Video Agent automates workflows in existing business software through its user interface. Our prototype creates invoices in IDURAR ERP without using application APIs, DOM selectors, or test IDs during invoice execution. Workflow recordings are processed by VAST’s video pipeline, using NVIDIA Cosmos for video descriptions. Retrieved clips provide evidence for reviewed, pinned execution plans. A conversational assistant collects invoice details, a typed Task API validates them, and a deterministic policy gateway requires approval for totals above $500. W&B-hosted zai-org/GLM-5.3-Flash reads fresh screenshots and locates controls. The execution engine uses Playwright mouse and keyboard actions, checking the visible state before advancing. The model supplies observations; the engine controls permitted actions. When an unexpected screen appears, execution pauses. Operator controls take over the same live browser session, resolve the interruption, and release it. The engine re-verifies the screen before resuming. Submission tracking prevents automatic repeat saves when the outcome is uncertain. Our demo creates a $634.70 draft invoice, demonstrates approval and an injected interruption, resumes after operator takeover, and verifies the saved record—with one Save action recorded.

You describe a scenario in plain words (e.g. "person close to a moving vehicle"). LongTail searches every camera separately in the VAST/VSS archive (Cosmos3-Reason descriptions, Cosmos Embed1 vectors, YOLO11 detections), then sends each hit to NVIDIA Nemotron 3.5 Lightning on W&B Inference for a yes/no/unsure verdict with a reason. The page shows verdicts live, plays confirmed clips and saves them as a test set. After each run, a second W&B call reads the rejection reasons, explains what the archive's descriptions fail to capture, and suggests a better Cosmos ingest prompt. Example: 22 search hits -> 1 confirmed (a person running toward a moving forklift); the descriptions never state distance between people and moving vehicles. Built solo with the Cursor Agent CLI and the event skills, deployed on CoreWeave Kubernetes; Weave traces verdicts from the CLI. Limits: verdicts use text descriptions, not pixels; borderline verdicts can change between runs; runs take 1-3 minutes.

Racoon uses AI to edit real-world footage. Upload any clip and it can remove, add, or replace anything in the scene, from a spill on the floor to a person in the frame, while keeping the original camera angle, lighting, and motion intact.

UnitProof is an AI inspector for property walkthrough videos. Owners define what gets checked per property (templates plus per-property policies): an apartment's oven and drains, or a workspace's 'no trash, walkways clear, appliances cleaned and work'. The agent skims the video, zooms into the seconds where something happens (temporal zoom), and judges every check with evidence anchored to the exact moment in the video. Function checks need cause and effect over time (the tap runs and drains, the burner ignites), which is why this is a video problem and not a photo problem. Where footage is not conclusive, it UnitProof is an AI inspector for property walkthrough videos. Owners define what gets checked per property (templates plus per-property policies): an apartment's oven and drains, or a workspace's 'no trash, walkways clear, red cap mandatory'. Someone films the space on their phone and says a one-time code out loud,asks for exactly the clip it needs; you film it and the check flips to verified. Every note links to the moment in the video and the report is shareable. Everything is mirrored live into VAST: clips into VAST S3 and verdict rows into VastDB, through a bridge running on the event lab VM. NVIDIA Cosmos-Reason2 in VAST's VSS pipeline gives a second opinion (it agreed on 4/6 checks in our live demo). C2PA provenance flags AI-generated or re-encoded uploads. In the demo we added a 'Red cap mandatory' rule live and the agent flagged the attendee wearing a blue cap.

Edge-Case Miner is a tool for robotics and self-driving teams that have hours of camera footage but no time to label it. You type a request like "forklift passing close to a person" and pick which cameras to look at. The tool searches the whole video archive for likely matches, then plays each candidate clip to a video-understanding model that answers yes or no and fills in labels such as who is in the shot, what they are doing, how close they are, and the lighting and weather. Clips that fail the check are rejected with a reason, so nothing gets into the dataset on a caption match alone. Every confirmed clip also gets a quality score based on object detection, blur, and whether the whole event is visible. At the end you get a zipped dataset with the clips, a labels file, and a dataset card, plus a coverage report that shows which situations you have plenty of and which you are missing, with a plan for what to film next. If too few clips pass, the tool proposes a better indexing prompt and, once you approve, re-indexes the closest chunks and searches again.

A lot of video footage is surprisingly static, and running a full interference sweep on unchanging frames is a waste of precious GPUs. We're optimizing the pipeline by using codec motion vectors, used for video codec compression, to crudely detect motion for free. This is a simple heuristic that gates a full YOLO run on the entire frame, saving up to 70% for low motion scenes, but could easily be extended to specific regions of the video and more expensive models. Everything uses my open source real-time media transport (moq.dev) allowing the AI model to only run on demand, saving even more money for rarely monitored security footage. YOLO is running on my desktop PC at home so I can process individual frames. It's connected to my global CDN (moq.pro) which it uses to both fetch the camera footage and publish the AI detection results (as a track). The web player subscribes to tracks on demand based on the camera selected and options. We emulate live footage using VAST's sample videos slowly tricked over the network (ffmpeg -re) and everything is real-time. Object detections and motion vectors (extracted by the desktop) trail the footage by at least 2 frames (plus RTT) because we intentionally don't synchronize the sources. Real-time latency is critical for some use-cases, like drones, so we felt it would make a more honest demo.

We built a carrier route optimizer that uses indexed video to tell which mailboxes have a deposit and which are empty. Carriers skip empty boxes and only stop where there is mail to drop. The result is a shorter daily route without guessing from the sidewalk.

test

Where should the bench go? Designers answer that by watching a plaza for weeks or scrubbing hours of footage, like William Whyte did. Most streets never get that attention, so seating, lighting and bus stop decisions get guessed. Linger is video agent that does this observation at scale. It scans ever camera in the VAST video library, ranks the places where people stop, wait and gather (linger), and lets a planner explore one: where people stop by day and night, what tends to happen next (a bus arrives, then a group forms where it stopped), and three plausible futures, like seating or lighting, each citing the real clips behind it. We re-ingested the footage with a behavior-only prompt, so NVIDIA Cosmos3-Reason never records faces, identity, or clothing. YOLO11 detects people and vehicles on CoreWeave GPUs. The agent reasons with W&B Inference, traces every step in Weave, and a Weave evaluation checks it never cites a missing clip or claims cause. Built for planners, transit agencies and campuses: weeks of observation in an afternoon, backed by evidence anyone can watch.

We use Nvidia Cosmos reasoner model and YOLO object detection to quickly detect objects. The Cosmos reasoner model then analyses the scene and then reasons out the whole scenario: whether there is a pothole, how deep the pothole is, which vehicle parts can be damaged because of this pothole, etc and then the whole thing is communicated to near by vehicles. We create a memory map of the whole city.

Someone almost gets hit today. They walk home, so it never becomes a report. Tomorrow it happens again, until one day it ends differently. 39,345 people died on US roads in 2024, and the warnings came first: near misses buried in hours of camera footage nobody watches. LookOut is an autonomous investigator for street cameras. Press one button and it: - Tracks every pedestrian, cyclist and vehicle with YOLO - Measures each interaction in seconds (post-encroachment time and time to collision) - Verifies every close call with NVIDIA Cosmos, rejecting false alarms and explaining what happened - Remembers everything in VAST, recalling similar events across cameras and time to prove the danger isn't isolated - Finds the recurring pattern with W&B and recommends FHWA Proven Safety Countermeasures, each linked to FHWA and the exact clips that justify it - Simulates the crash that almost happened: drag the WHAT-IF slider and see what 0.7 seconds of luck really meant Every step is traced and evaluated in W&B Weave, and every claim links back to video. Recommendations are for engineers to review, not automatic decisions. Crash reports show where people got hurt. LookOut shows where they almost did, so the next one never has to.

Robotics research needs datasets with verifiable diversity across environments, objects and actions. **MATCHA** helps **researchers** turn video archives into **curated**, **labeled datasets** for **training** and **evaluation**. Researchers describe the footage they need in plain language, inspect matching clips and objects at scale, identify coverage gaps and overrepresented scenarios, and edit labels in bulk. By making dataset composition visible and measurable, MATCHA helps teams build balanced datasets and export versioned collections with clear annotations and documented coverage. app password is: theyarecoming

LoveCheck is a three-site safety board: warehouse, neighborhood, and dashcam. It looks at those cameras and shows a few clips where someone might be at risk. The UI is Next.js. Flask talks to team Video Search with credentials kept on the server, so the browser never sees login tokens. It finds the latest footage, searches for hazards, drops near-duplicate chunks, and streams the clip through our own proxy.

Self-driving cars and warehouse robots rarely fail on everyday scenes. They fail on rare moments like a pedestrian stepping out from behind a van or a worker beside a moving forklift, and those moments are buried in hours of unwatched video. Longtail is a video agent that finds them. You describe a corner case in plain English, and the agent: - expands it into several searches - searches the indexed footage three ways: VAST's hybrid search, our own Cosmos-Embed index, and YOLO object detections - has NVIDIA Cosmos3-Reason watch the best candidates and keep only real matches Results are ranked by danger (Cosmos's rating) and rarity (embedding distance within each camera pack). Each clip comes with a structured scenario label, "find similar" turns one example into a set, and a coverage map shows which situations are missing and lets the agent fill those gaps. Every run exports a training-ready JSON with clip IDs and start/end times, or a versioned Weights & Biases dataset. It works across real dashcam, highway and street cameras and synthetic warehouse footage.

Hype Agent is a video agent that turns a raw, wide-angle soccer clip into a highlight people want to watch. The clip is ingested on VAST AI OS and indexed with NVIDIA's VSS blueprint, which gives YOLO detections on every frame, and Cosmos Reason reads the frames to find the story. In our demo clip, Duke's #8 beats a defender one-on-one, strikes on frame 220 and scores past a diving keeper. The agent tracks the ball on every frame and writes an edit plan: speed ramps, freeze frames, slow motion, camera zooms and tilts, fire trails pinned to the real ball, player tags, text slams, and sound effects synthesized from code and timed to the action. It renders the cut, re-watches it frame by frame along with the audio waveform, critiques what's weak (a slow opening, a muddy picture, dead air, cluttered text), and revises the plan. After five loops a flat clip becomes a 19-second edit with a cold open, a slow-motion replay and audio mastered to social-media loudness, with no human editor involved.

A small robot mouse with a camera which is rightfully terrified of cats. It patrols the room. When NVIDIA Cosmos Reason (cosmos3-nano-reasoner, served on CoreWeave GPUs) sees a cat in the live feed, the car panics: red lights, a 540 degree spin and a sprint away, with a panic clip on the dashboard. When Cosmos sees a hot dog, the car flashes green, and approaches. Every 5-second clip the robot records goes into the event's VAST AI OS video pipeline, where DataEngine runs YOLO11, Cosmos captions and Cosmos-Embed1 into VastDB (756 clips so far). So the robot can answer plain-language questions about its own life: "when did the cat get closest, and did I get away?" comes back with an answer and the matching clips. A Llama 3.3 70B planner on WandB Inference narrates what the robot is doing. Reflexes (steering, a 25 cm wall check, a disconnect watchdog) run on the laptop and never wait on the cloud; YOLO11 is the offline fallback detector. For anyone with a pet camera, a security feed or a robot: video that can be searched and acted on instead of sitting unwatched.