Differential Swarm - Self Improving Security Evals

Differential Swarm tests how an AI agent’s behavior changes with every pull request. It maps the change’s blast radius, generates targeted attacks, and sends a swarm of red-team agents (scam callers, manipulative sellers, poisoned listings) against the old and new builds in isolated paired sandboxes. Comparing each pair shows regressions, fixes and pre-existing failures. Our demo uses a shopping agent facing fraudulent voice calls, manipulative sellers and malicious listings. A live dashboard shows the swarm’s progress and lets reviewers inspect conversations, tool calls and payment evidence side by side. Each finding becomes a GitHub remediation issue and a verified Semgrep rule, which turns behavioral discoveries into reusable static checks. That’s how the evals improve themselves.