Developer Projects

**Problem**. Delivery robots get stuck at crosswalks: they spin in place, miss the walk phase, or start into traffic. **What it does**. CrossWise watches street video and decides GO, WAIT or NO-GO every 0.2 seconds, with the evidence drawn on the video. When a robot would wait too long, it escalates to a reroute or a remote operator. A scorecard ranks 7 crossings by difficulty for route planning. **How it works**. Vehicles are tracked from the YOLO11 boxes VAST stores for every frame, with camera-shake removal and 1.5-second path prediction. The pedestrian signal is read from pixels. A vision model on W&B Inference finds cones, barriers and the free path. Unclear evidence forces WAIT. NVIDIA Cosmos Reason captions and Cosmos Embed search, on CoreWeave GPUs, found the crossing footage in the VAST archive. Decisions are stored in VastDB and model calls are traced in W&B Weave. **What we learned**. Our first version had an LLM judge 5-second captions. It confused the car light with the pedestrian signal and parked cars with traffic. We rebuilt the core as frame-level perception and measured both on 5 clips: starts against DON'T WALK fell from 3 to 0, crossings rose from 6 to 8, and time from WALK to rolling dropped from 3.4 s to 1.0 s. Limits. Decision support on recorded footage; it does not control a robot. Built end to end with Cursor.

We built an ESPN style live sports and events broadcasting studio which uses the VAST, NVIDIA, YOLO and CoreWeave stack for automated directing, switching the camera, creating replays and adding overlays and commentary to give a professional experience. Think of it as your personal camera van for any event you want to have produced professionally- now possible in seconds and at a fraction of the cost.

Antigense Daisy is an evidence-driven agentic cyberdefense prototype. A controlled worker-byte fault exposes a fail-open authorization fallback. Semgrep CE detects the pinned flaw (1 finding before; 0 after). A four-case regression oracle confirms the correction denies unauthorized work while preserving legitimate requests; AI-generated advice never authorizes its own application. Each recorded transition becomes an addressable Fractal Custody Object (FCO) linked through typed Fractal Custody Graph (FCG) relationships. Ordered Merkle/MMR checkpoints support independent replay and byte-integrity verification. A separate 12-leaf local review execution is browser-verifiable. Executed sponsor evidence: Semgrep local scans; Akash GPU deployment and Qwen2.5:0.5b inference (HTTP 200, 3,466.4 ms), followed by a successful close request; and ClickHouse Cloud ingestion/replay with 55 exact historical readbacks. An earlier Akash HTTP 401 failure remains preserved. Successor 050 adds a live LOCAL judge cockpit: ALLOW/DENY reviews append FCOs/MMR roots; spectators receive redacted SSE updates; private notes require local authorization. All 17 local tests passed, including unauthorized, cross-origin, stale-root and tamper controls. Local 050 is NOT remotely hosted and does NOT re-run sponsor cloud APIs. Judge identity, hardware/TEE attestation, Pi execution, real payments and financial ROI are unverified. Hashes establish byte integrity, not truth or causality.

**Great stories are hiding in messy footage. **VGA brings them into play with two creative teammates: a Context Agent that discovers meaningful moments and connects them into a story, and a Video Agent that turns that story into a video. Wrapped in a retro game-console experience, VGA makes the journey from scattered clips to a finished story feel like a creative quest. **Press Start. Find your story. Bring it to life.**

Aloha Circle turns a Maui vacation into a relationship with Maui. From the moment visitors land at Kahului Airport, a QR banner at baggage claim starts their aloha with a donation to a live local cause; at the physical Aloha Circle kiosk, keiki and adults complete the Breath of Aloha — a camera-tracked ritual (honi ihu forehead hold, eyes, ears, nose, heart) guided by Kanaloa, a lifelike water-spirit avatar — then spin the Wheel of Aloha to win a sponsor-donated experience and meet the Maui local whose style best matches theirs, claimed instantly by QR or email. Behind it, the Aloha Agent continuously ingests a Maui Needs Index of local causes and recomputes matches between travelers, locals, nonprofits, and donated experiences, so every visit ends in a real action: money to a cause, and a traveler and a local actually meeting. Sponsors are first-class citizens: local businesses donate the experiences on the wheel (helicopter tours, dinners, excursions), get brand placement in the kiosk and on aloha-circle.com, and a sponsor VLM even reads visitor brand/style signals to drive the match. On the tech side the build is sponsored/powered by VAST Data (S3 + VSS video search), NVIDIA (Cosmos Reason / Cosmos Embed NIMs), W&B Inference by CoreWeave, Roboflow (RF-DETR), and Masky avatars — built across the Agent Harness Hackathon (TrueForge × Qodo × Bright Data) and the VAST Builders Challenge.

Yield or proceed, use cosmos reason to label warehouse footage indexed on VAST and train a tiny CPU model on YOLO geometry to make the same yield-or-proceed call a robot would

Property Studio helps realtors turn a listing into an evidence-backed video script. It combines photo analysis, property research, and editable financial scenarios to create compelling stories that buyers and sellers can investigate.

Sidewalk Watch uses street-level video to automatically detect temporary sidewalk obstructions that can create accessibility barriers, such as parked vehicles, construction equipment, trash bins, and other objects blocking pedestrian paths. The system analyzes video segments using computer vision and video-reasoning models, identifies potential obstructions, and attaches location and timestamp information so barriers can be mapped and reviewed. The goal is to complement projects like Project Sidewalk by detecting what is blocking a sidewalk right now, rather than primarily documenting permanent accessibility infrastructure.

**The problem. San Francisco pledged Vision Zero in 2014, yet the first half of 2026 saw 100 fatal or severe crashes, more than the 92 in 2014. The city's new Street Safety Act asks for quarterly progress dashboards. Street cameras already record the close calls, but nobody has time to watch them.** ** ** **What it does. ViZ Agent searches video indexed on VAST (NVIDIA VSS retrieval) for four kinds of pedestrian–vehicle conflict across a Toronto dashcam and four San Francisco street cameras. Each candidate must pass three independent checks: YOLO detections, the Cosmos Reason caption verdict, and a Nemotron grader on W&B Inference. Cosmos Reason can take a second look and explain what physically happened. Verified clips are ranked into hotspots, and the agent drafts an Open311 work order with an FHWA proven safety countermeasure picked from the conflict's cause.** ** ** **For whom. City traffic engineers and Vision Zero teams. In our live run, 60 candidate clips became 4 verified conflicts, so an engineer reviews 4 clips, not 60.** ** ** **Human in the loop. An engineer approves or rejects each clip with one click. Those labels measure the agent's precision and are published to W&B Weave as a dataset and an evaluation. Every search, check and model call is traced in Weave, and nothing is sent without sign-off. The whole sweep is also packaged as an Agent Skill.**

**We built on the VAST Builders Challenge stack: neighborhood and street cameras already indexed in our team VSS archive. Search uses Cosmos Embed hybrid retrieval over segment captions; Cosmos Reason supplies the descriptions we search and summarize; YOLO detections help filter vehicle-related moments. Cursor skills drove login, search, agent Q&A, and optional re-ingest when we needed captions tuned to parked cars and curb activity. App logic runs on top of the VSS APIs so the demo is search → evidence clip → answer, not a rebuilt ingest pipeline.**

OriginX turns plain-English safety rules into alerts over camera footage. It replays the indexed archive as live camera feeds, filters segments using YOLO detections and captions, and an LLM judge raises scored incidents with written reports. Built for site safety leads.

Dashcam and fleet archives often have no GPS. VAST VSS finds clips by meaning, but its location filter only says "Toronto". Evidence Atlas answers where. - Cosmos Reason re-watched all 1,080 indexed Toronto clips and read 3,658 street signs and business names. - Names are matched in OpenStreetMap, never model-guessed coordinates. Two corner signs that meet give a verified intersection (~60 m), a unique venue gives a landmark (~150 m), and guide signs give an area. - Drives are continuous, so a route check rejects matches the car couldn't have reached. For example, an "Art Gallery of Ontario" banner read 5 s after the ROM, 1.6 km away, is rejected. 86 clips are placed, 18 are rejected, and 831 are left off the map rather than guessed. - Every pin opens an evidence chain: the video, what was read, the OpenStreetMap match, what agrees or conflicts, and where the index caption disagrees. For whom: six built-in use cases, each with its own report written from saved evidence only by Llama 3.3 70B on W&B Inference: ad and retail site scouting, insurance and fleet incident location, roadwork audits, curb and delivery management, cycling and pedestrian safety, and newsroom footage verification. Places, not people: nothing identifies faces or licence plates.

cross walk watcher for padestrian

Robotics and AV teams own thousands of hours of footage nobody watched. Grabbit Mining Co. turns one sentence ("forklift within 2 m of a worker in an aisle") into a human-verified test set of edge cases. How it works (every step runs on the sponsor stack): 01 Expand — W&B Inference rewrites the sentence into 3 query variants (traced in Weave). 02 Search — Cosmos Embed vectors in VastDB over 2,352 segments indexed by VAST DataEngine; all cameras, deduped, top 12. 03 Evidence — Cosmos Reason captions + YOLO11 detections from the VSS pipeline on CoreWeave GPUs. We re-ingested 8 warehouse chunks with a custom RISK prompt (distance to forklift, walkway, running) because the default captions lacked those facts. 04 Severity — W&B Inference returns strict JSON {is_match, severity 1-5, tags, rationale}; median 402 ms per clip. 05 Review — a human presses Confirm or Reject on the page. Grabbit can't reach its own button; that's the safety feature. 06 Act — export test set (JSONL) for policy eval. Key finding: the only severity-5 clip for the forklift query sat at search rank 9, and at rank 7 for person-vs-vehicle. Similarity search alone would have buried both. You can't just search; you have to mine. Today's run: 3 queries, 36 hits, 17 flagged, 15 human-reviewed, 60% confirmed, 9 in the test set. Built in Cursor with its VAST skills. The page is a static replay of today's run so nothing breaks on stage; the pipeline is live on the VM (mine.py).

**Project description:** ** ** **We built Near-Miss Board, a semantic search console for traffic safety footage. Type a plain-English query like "a car driving on highway" or "truck changing lanes on the highway" and get back timestamped moments with snapshots and AI summaries across every camera in the archive.** ** ** **How it works: the Nashville I-24 multi-camera corpus is ingested and captioned through VAST's video search stack, turning raw video into a semantically searchable index of moments. The board queries that index live, ranks matches by similarity, and serves the clips with per-moment summaries. It is fully deployed on our team's cluster namespace behind the event ingress at team-16-app.thecosmoslabs.com/app.** ** ** **Why it matters: near-misses are the leading indicator of fatal crashes, but finding them today means scrubbing thousands of hours of video by hand. Near-Miss Board turns that into a five-second search, so traffic engineers can spot dangerous corridors and fix them before someone gets hurt.**

sadcas

Warehouse Safety Monitor turns warehouse camera footage into a ranked queue of near misses, blocked aisles, crowding and unsafe behavior. Each incident shows the clip, measured distances and a suggested fix. Hotspot heatmaps and an end-of-shift report help supervisors act.

StreetStyle Scout makes street footage searchable by visible clothing and colors. Users enter a plain-language query, browse matching frames, and play the source video to check the result. VAST provides video retrieval, YOLO locates people, and NVIDIA Cosmos analyzes visible clothing. The workshop’s models run on CoreWeave GPUs. Cursor assisted backend development, and Claude Code built the frontend. Results separate observed details from uncertainty and link back to video evidence. The prototype focuses on selected San Francisco clips and requires human review because search relevance is imperfect. It does not identify people, guess clothing brands, or infer citywide trends.

PERJURYEvery claim about the footage takes the stand. Say or type a sentence about a traffic video. PERJURY splits it into atomic claims and sends each one to the cheapest witness that can settle it. Each claim is ruled SUPPORTED, CONTRADICTED or "the pixels can't tell", with the evidence. The Cosmos jurors never see the claim, and code computes the verdict, not an LLM. It is a verify(claim) tool for any person or agent that has to sign off on what footage shows. How it uses the stack:**VastDB:** the records witness. Pushdown queries settle classes, counts and metadata across every segment before any GPU call. **S3:** keyframes and 4K evidence crops. **DataEngine + VSS:** reads the ingest pipeline's output, and evidence reels go back in via videos/upload. The stock agent is the A/B baseline. **Cosmos3 Reasoner:** jurors answer fixed questions on keyframe grids and put a box around each "yes". **Cosmos-Embed1:** picks the most claim-like moment per camera. **YOLO11:** its detections feed the records, and a zoom check must confirm every Cosmos "yes". **Canary-1B:** speech-to-text for spoken claims. **Nemotron on W&B Inference:** splits sentences into claims and labels captions. **Weave:** traces every verdict, logs 👍/👎 feedback, and runs the 47-claim bench. **CoreWeave:** GPU host. Each verdict shows its GPU-seconds. **K8s + FastAPI:** live at /app. **Cursor:** ships a perjury-verify skill.

Integrating Andon with Vision Training to Develop Safe Workflows

VAST CAM connects video data extraction with camera-specific model training. A dataset agent queries VAST AI OS, reviews video evidence, and extracts short pre-event clips with boolean labels indicating whether a person–robot or person–forklift collision follows. We then use this dataset for supervised fine-tuning of Qwen3.6-27B. The goal is to turn a short clip from a fixed warehouse camera into a probability of collision within the next few seconds. The model sees only the lead-up footage; future footage is used solely to determine the training label.

A review assistant built on the supplied VSS archive. A reviewer searches indexed footage in plain language and gets ranked evidence cards linked to the source interval. A W&B-hosted LLM assesses each card against a fixed event definition and rule card. The reviewer then records a decision, which is stored separately from the machine assessment. Live: https://team-6-vss.thecosmoslabs.com/app

**WORLDSTATE turns video from passive footage into operational memory.** Instead of only detecting objects or describing individual frames, WORLDSTATE watches repeated executions of a physical process and learns the normal sequence of states and transitions without manually labeling each step. In our robot pick-and-place demo, WORLDSTATE learns the normal process: approach, grasp, lift, carry, descend, and place. When an object unexpectedly shifts sideways, the system detects the exact moment reality diverges from what it learned — at 1.904 seconds — and flags it as a novel failure. NVIDIA Cosmos then explains what physically happened in natural language, while YOLO tracks the objects involved. The operator can click **Remember This Failure**, adding the event to WORLDSTATE's memory. When a different run later fails in the same way, it is immediately recognized as a **known failure** instead of an unknown anomaly. WORLDSTATE also includes searchable event memory and a 3D digital twin for visualizing failures and simulated recovery paths. We additionally integrated our team's real VAST warehouse corpus: 40 clips with Cosmos Reason captions and YOLO11 detections that can be searched directly from the WORLDSTATE interface. The goal is simple: give robots, factories, and warehouses a video agent that learns what should happen — and remembers when reality breaks.