Developer Projects

NYC DOT’s traffic cameras watch the city’s most dangerous corners but record nothing, so every near miss is lost. PharOS is a 3D map of Manhattan with a heatmap of injury crashes from the last 12 months of NYC Open Data, and every node is a real DOT camera. Open an intersection to see its camera image, crash history, reported causes, crashes by hour, and next steps. For continuous video, PharOS monitors live 511NY feeds and returns a verdict on each 8-second clip in about 14 seconds. Plain-English search finds matching moments in recorded footage. How we used the stack: VAST AI OS: The workshop’s VAST pipeline had already indexed recorded New York footage into VastDB. PharOS queries it with hybrid semantic search and streams matching clips for seekable playback. NVIDIA: On each live clip, YOLO11 boxes every car, person and bike across 32 frames, while Cosmos Reason reads the same video and describes what happened. Neither can claim motion from a single frame. CoreWeave: Its GPUs serve the NVIDIA models, which return results in under 3 seconds. Weights & Biases: W&B Inference runs Llama 3.3 70B. It turns each clip’s findings into a structured verdict, turns questions into search plans, and writes next steps linked to official guidance. Next: live video at every DOT camera, and a citywide ranking of intersections by near misses for planners.

**Shepherd: an AI night watch for small farms** Every video agent watches people, cars or forklifts. Nobody is watching the animals. Shepherd reads every clip a farm’s motion camera records and tells the farmer which ones matter, so no animal gets hurt because nobody was watching. We ran it on 16 real clips from our own chicken coop in Fredon, NJ and build this app.

FightLens is an app that helps viewers understand what is happening in boxing and MMA, and why the balance of a fight is changing. It brings fighter tracking, exchange analysis, and win-probability estimates into one interface, with corresponding video moments viewers can revisit to inspect the evidence. Our event-driven pipeline separates fast action detection, contextual video understanding, and decision updates. YOLO acts as an always-on sentinel, tracking fighters and flagging potential attacks. When an exchange is detected, NVIDIA Cosmos receives a clip with context before and after the trigger to interpret the action and generate an explanation. This design aims to reduce redundant video-model calls without missing brief exchanges. VAST Data organizes video features, real-time events, and related data into shared context for prediction. The Jev architecture with clef-flash supports rapid updates to the estimated advantage. Each exchange, explanation, and probability update is linked so viewers can follow the reasoning. The displayed probabilities are prototype estimates, not yet validated or calibrated. Next, we plan to use historical fight videos to evaluate and improve the workflow, measuring end-to-end latency and prediction quality separately. Our team combines boxing experience, AI research, and product development to make combat sports easier to follow.

When a car flees a crash or a person goes missing, operators still click from feed to feed by hand, guessing which camera comes next. The footage exists. What's missing is something that can follow one target across hundreds of feeds without getting tired. We predict where the suspect can go and try to provide insights to police while chasing

CarWitness (车证) is "Ctrl+F for street cameras." A live event feed flags vehicle events across every camera, and anyone can ask in English or Chinese to get matching clips with NVIDIA Cosmos descriptions, YOLO11 detections and playable video. A language model on W&B Inference plans each search. Selected clips become a bilingual incident report where every finding cites a clip timestamp and unsupported findings are dropped, and any clip can be exported as an AV-simulation scene card. The UI switches between English and Chinese.

Spatial Video Agent answers one plain-English question across every camera in the corpus: 3,537 clips from warehouses, a Toronto dashcam, and New York and San Francisco street corners. For each hit, it runs tracking and geometry at query time and returns an answer card grouped by evidence level. Where a camera has calibration or GPS, the card shows the person-to-vehicle distance in meters with an uncertainty. Where it has neither, the agent abstains and says “no geometry”. Video and detections live in VAST S3, and captions and Cosmos embeddings are read live from VastDB on every query. NVIDIA Cosmos 3 Reason and Cosmos-Embed1 run on CoreWeave GPUs. Our FastAPI app runs on the CoreWeave cluster. W&B Weave traces every agent step. Every number comes from CV and geometry, not from the model.

StreetCast turns hours of street video into a cited record of what's blocking New York's streets, and helps people act on it. For riders (Ride report): a delivery cyclist records her ride. StreetCast replays it and every problem lands as a field note: blocked bike lanes, double parking, trash, obstructions, graffiti, construction. Each one comes with a drafted 311 complaint: category, street, what happened, the company and fleet number, and a link to the exact clip. It also shows how much time she lost and what caused it. Fleet numbers, never license plates: accountability for companies, privacy for people. For newsrooms (Your Block): a producer picks a neighborhood, a time of day and a travel mode. StreetCast runs six searches across the footage and builds a one-minute narrated segment. Every sentence cites the clip and time range it came from. No clip, no sentence. Eval: claims are checked against the clips they cite, and we report the grounding rate. How it works: VAST stores, indexes and searches the footage in VastDB. NVIDIA Cosmos Reason describes every clip, Cosmos Embed powers the search, and YOLO11 detects objects. An LLM on W&B Inference turns those descriptions into structured events and scripts, with every call traced in Weave. Built with Cursor. Rider files it. The city sees it. The newsroom airs it.

SafeRoute is an AI-powered evacuation planning prototype that demonstrates how street-camera evidence can inform routing during disasters. It combines real archived NYC footage with simulated wildfire and hurricane effects, uses NVIDIA Cosmos to assess visible road conditions, and calculates routes around blocked or potentially dangerous intersections. Built with VAST’s video archive and W&B inference and tracing, it provides an interactive map, evolving disaster scenarios, and plain-language route briefings. The project is an evacuation drill, not a live emergency navigation system

World Compiler turns captured video and sensor data from smart glasses into a structured, inspectable model of the physical world. It organizes observations into objects and claims, links each interpretation to its original evidence, and preserves a revision history. Users can review uncertain results, correct interpretations, and replay how the system’s understanding evolved without changing the original recordings. Our hackathon work focused on building the actual compilers and connecting them to **World Sensor**, our separate capture platform for phone, desktop, and watch. The technology stack brings together: **Cursor** for developing the compiler backend, frontend, and integration workflows.**VAST Data:** Provided the VM used to run Cursor’s remote development and build environment for the compiler project.**CoreWeave** as part of the cloud compute infrastructure for GPU workloads.**YOLO** as the object-detection component of the perception architecture, feeding observations into the compiler rather than treating detections as final truth.The key distinction is traceability: the system connects what it believes about the world to what it actually observed, with human review and recorded corrections.

**Witness - Evidence Over Assumptions**** **When drivers provide conflicting accounts of an incident, investigators need a reliable way to examine what video footage actually shows. Manually reviewing recordings, locating important moments, and comparing statements can be time-consuming. Witness is an AI-assisted video evidence review platform designed to make that process faster, more transparent, and easier to audit. **What we built:** Witness provide a web-based investigation workspace featuring: 1. Video evidence playback: Stream footage from VAST storage and navigate the recording using clickable, timestamped evidence markers. 2. Claim verification: Compare individual statements with available video observations and classify each as Supported, Contradicted, or Not Visible 3. NVIDIA Cosmos reasoning: Identify vehicles, pedestrians, traffic signals, and other visible objects as additional context. 4. YOLO11 object detection: This identifies the actual vehicles, pedestrians, traffic signals, and other visible objects as additional context. 5. Evidence inspection: Present claim-specific timestamps, supporting observations, contradiction cues, and explanations in a unified interface. 6. Human Oversight: Clearly distinguish observed evidence from AI-generated interpretations. Witness does not automatically determine legal fault or liability.

CloseCall finds **near misses** in the street video cities already have. NVIDIA VSS searches the VAST library (3,537 clips, 13 cameras), YOLO11 boxes every car, bike and person and uses their real size to measure gaps in metres, and NVIDIA Cosmos describes each clip so we can **flag hazards like a car in the crosswalk**. Each clip gets a **0 to 10 danger score**, places are ranked as hotspots, and a person approves every finding against its clip.

Traditional computer vision systems can identify objects but understanding when their presence is relevant often requires some custom programming for each application. Occlusive Logic bridges that gap by allowing users to turn existing video detections into configurable, actionable rules. Users select a video source, draw a polygonal zone directly over the footage, choose the object classes, and configure confitions such as entering an area, remaining there for a specified duration, or being absent. A custom Python rules engine evaluates the object positions, confidence thresholds, the tracking informations and temporal conditions to determine when a rule has been satisfied.  When triggered, the system creates a time stamped incident ticket with visual evidence. Users can inspect, revisit, and resolve incidents through a web dashboard.  The application integrates with VAST's video infrastructure and uses YOLO detection metadata from VAST provided footage. It is built with Python, FastAPI, Javascript, SQLite, and a custom geometry and temporal state engine. Additional simulations demonstrate potential applications in accessibility focused occlusion monitoring and in proper waste management.  Computer Engineering foundation: Occlusive logic treats camera detections as virtual sensor signals and applies programmable spatial and sequential logic to produce an event. This lays a foundation for future boolean rule composition and edge hardware deployment.

Drive Recap turns unused dashcam video into a trip report you can actually read. We used team already-indexed Toronto footage (pie_cam-3, set01): 180 segments with NVIDIA Cosmos captions and YOLO detections on VAST / CoreWeave. We do not re-ingest or call those models ourselves. A tagger turns captions + detections into moments (yielded to a pedestrian, cyclist shared the road, red-light stop). The Awareness score is yielded / segments with a pedestrian in the crosswalk (7 of 16 on this drive = 44). It is not a safety score or a hard-brake detector. Counts are estimates. The app shows the score, four stats, and playable clips (proxied so credentials never hit the browser). Ask this drive searches driver-specific questions (e.g. all red-light stops) and returns clips with timestamps.

**Problem.** In sports with judges, calls happen in a split second and people argue about them all the time. Friends and parents film every game, but those videos just sit in our camera rolls. We started with kendo because we both do it: a point only counts if the shout, the strike and the stomp all land in the same quarter-second. **What it does**. Drop in a match video and SecondCall judges each strike: POINT, NO POINT or CAN'T TELL, with how sure it is and the proof. It also catches penalties (hansoku) and points the referees might have missed. How it works. First it listens: the stomp and the shinai crack tell us exactly when the strike happened, and NVIDIA Canary hears which target was called. YOLO11 zooms in on the two players and blocks out the referees, then NVIDIA Cosmos looks at the frames around that moment. A rule card turns all that into a verdict, W&B Inference explains it in plain words, and Weave tracks every step. The app never sees the referees' flags, so it can't be swayed by them. Rulings live in VastDB, matches are searchable in VAST Video Search & Summary, and it all runs on CoreWeave GPUs. Beyond kendo. Every sport is just a new rule card, so it works for judo, fencing, pickleball line calls, or any sport where calls get argued. What we learned. Plain 5-second clips kept missing the strike. Listening for the moment first, then showing Cosmos close-up frames, fixed it. Limits. It's a second opinion, not a replacement for referees. Built with Cursor.

Witness desk is for someone who saw a vehicle and needs the footage. They choose the city, an optional time, the body type, and the color. The desk searches the indexed archive and lays the closest clips out as a network. The lit node plays in place. Yes keeps a still with the case. No clears that clip and moves to the next. The search uses the stored clip descriptions from the video stack. The desk shows those captions and does not invent events they do not contain. Plates and make or model are not in this index, so the form does not ask for them. Stolen and hit-and-run are marks a person sets after watching a clip. They are not part of the query. CoreWeave hosts the video models inside that search stack. It shows the captions those models already wrote. Cases stay in the browser, and the live site needs no login.

This is a dating app for vehicles. We take street camera footage, detct and track the vehicles in it and give each one profile built from it how it behaves. Compatibility is computed on the video not just narrated by a LLM. We score things on what the footage shows: speed and driving lanes, times of day etc. The LLM explains the score instead of inventing it so every claim traces back to a clip **Ingest:** Street footage is loaded and indexed so it can be searched and streamed by timestamp **Track:** YOLO detects and tracks vehicles frame to frame, producing a track history for each one (path, speed, dwell time, proximity to others). **Understand:** Cosmos describes each vehicle’s behavior and its interactions with other vehicles (“yields to cyclists,” “idles at the curb”). **Profile:** An LLM turns the track data and Cosmos descriptions into a bio, a nickname, and personality traits **Match:** A scoring function compares track features for each pair of vehicles. Pairs that shared a clip or camera are candidates **Explain and chat:** An LLM writes the compatibility breakdown and a “first date,” and vehicle personas chat with their matches and accept or reject based on their profiles VAST  We read indexed data, stream the clips and persists complaints and tickets Nvidia - Cosmos reasons captions aree the evidence. YOLO waks the judge cosmos embed runs search and interview Cursor The whole project was built using it Weight Biases gpt-oss-20b judges the clip. Llama writes it.

An application that uses VAST video data to identify any clips where broken asphalt, blocked bike lanes, broken streets, potholes, or damaging areas potentially for cyclists exist. The goal would be to gather a confidence score from users on how many people requested a fix for these areas, looking for a time interval between the complaints and notifying both ends to see if any users raise the confidence score high enough for the DOT to receive a message. When the process is completed, both the users and the offices both receive messages that the problem has been resolved.

**CrossWatch** is a “jaywalk case agent” app. It sorts illegal pedestrian crossings into types, confirms them with YOLO object detection on the video clips, searches an archive for similar footage, and recommends street design fixes.

The problem At a crossing, a person on a bike needs to see who is about to step out, early enough to stop. Hedges, planters, kiosks and parked cars set right up to the crossing take that view away. Cities fix this by daylighting, clearing the space before crossings, but finding the bad corners takes site visits and complaints, and those checks happen from a driver’s or walker’s eye, not the bike lane. What it does UnfoldUnrest turns ordinary first-person ride video into that audit. It rebuilds the street in 3D, finds the crossings and everything around them, and asks one question at each: could the rider see both sides in time? Where they couldn’t, it names what was in the way, suggests the fix, and shows the frames that prove it. The value Every ride becomes a survey. Planners get specific corners and objects to act on instead of a general call for safer streets. Advocates get evidence they can point to. Riders learn which crossings to slow down for. It needs only a camera, so it reaches any street people ride. How the sponsor tools fitW&B Inference runs a vision model as an independent second opinion: the 3D check makes a claim, and the model looks at the real frames and confirms or rejects it, so whoever acts on a finding can trust it. Weave traces every stage, prompt, image and answer, so each finding can be followed back to its footage and reasoning. Clips arrive from the VAST environment carrying their source, so every result traces to the video it came from.

For warehouse safety managers and operations leads: the cameras already record every shift, but nobody has time to watch them. Real time: every 30 s the agent checks what VSS has indexed, analyzes new footage once it is fully indexed, and the app updates within seconds. Safety you can trust: motion tracks from the YOLO detections propose candidate near misses, NVIDIA Cosmos Reason checks that exact time window and image region, and the camera views vote. Asked directly, Cosmos called 168 of 180 clips a near miss, so the agent never relies on that question. With verification, 0 of 12 random control moments were flagged and all 3 drill near misses were caught. Operations: idle labor, idle machines, congestion and underused zones, with recommendations tied to the evidence and an idle-cost estimate with editable assumptions (about $859 recoverable per shift, about $215k a year at the defaults). Ask and report: plain-English questions answered by NVIDIA Nemotron-3-Ultra on W&B Inference, grounded in the analyzer's facts and VSS search, citing alerts and clips. Every number in the shift report is checked against the data. Results go back to VastDB (alerts, flags, per-segment metrics), keyed by each segment's S3 URI, so they join with the VSS captions and embeddings. Built with NVIDIA Cosmos Reason and Nemotron, VAST S3, DataEngine and VastDB, CoreWeave GPUs, W&B Inference and Cursor, on 60 videos from 33 camera views.

Video AI fails quietly on conditions your cameras don’t see often. Spore finds those gaps, grows synthetic data to cover them, and turns that data into evals for training better models. Spore searched VAST’s video index for missing conditions: the highway cameras have 30 clips, mostly clear daylight, and none at night, in rain, in heavy fog or in snow. An LLM on W&B Inference, traced in Weave, ranks which gaps matter. Spore grows the missing weather onto real clips with a physics weather layer and NVIDIA Cosmos Transfer. The cars stay in place, so the labels carry over and testing runs automatically. On CoreWeave GPUs, it tests the models in the VSS pipeline: in light fog, YOLO finds only four in ten visible cars, while Cosmos Reason keeps counting them. The result is a new benchmark and training set for better models.

Stores count foot traffic, but they can't see what the street is wearing. AURAAFIT reads outfits from video. It starts from event footage indexed in VAST VSS (612 clips, 13 cameras). NVIDIA YOLO11 finds and tracks each person, and W&B Inference vision (Gemma 4) labels what they wear and carry: top, bottom, outer layer, bags. Every labelling call is traced in W&B Weave. Requiring a real garment type in our prompt took typed labels from 49 to 213 out of 214. NVIDIA Cosmos3-Reason adds a scene caption for each 5-second piece. The app at auraafit.tech replays recorded street cameras from San Francisco and New York with the labels on the boxes. Buyers can search the labels ("backpack"), and each match links to its clip and timestamp in VAST. A dashboard rolls up 214 person-sightings into colour mix, style mix and carried items, with CSV export. A merchandising agent reads only the totals and drafts store actions, such as putting carry accessories near the entrance because 32% of sightings carry a backpack. Privacy by design: a guard filter keeps clothing and carried items only. Age, gender, race and faces are never inferred, and nothing is tracked across cameras. Counts are sightings, not unique people. Built for retailers, landlords and business districts deciding what to stock or lease on a block. Next step: a consenting storefront camera, over repeated days.

Nostradamus turns an indexed archive of video footage into a live prediction market. Each round plays one 5-second video segment and asks a single question about the next five seconds, rotating between pedestrians, vehicles, and intersections. A Weights & Biases model reads only the current clip's caption and quotes an opening price in cents, with its reasoning shown. You buy Yes or No at that price, choosing how many contracts, then settle: the following indexed segment plays, W&B judges it from that caption alone, and your buying power moves by M$1 per contract. The same question is listed as a real play-money binary on Manifold, opened at the model's price, bet on your behalf, and resolved from the clip.