Honeypot generator for RL environments

The recent OpenAI + HuggingFace incident reveals the dangers of reward hacking during post-training. Models with misaligned behavior may collude and hack into services to complete their tasks. As more SMB's are beginning to finetune their own models using open-weights models, it is critical that folks have confidence in their agent. But how can we detect models with misaligned behavior? One option is to generate honeypots. These are RL environments that cannot be solved by an LLM without performing misaligned behaviors. The only way that a model can complete the task is by performing an exploit. Models that succeed in a honeypot environment are necessarily dangerous.  Our tool takes existing RL environments on HuggingFace and mutates them to generate honeypot tasks. We use SemGrep to detect vulnerabilities to (1) find candidate RL environments and (2) confirm that our honeypots are exploitable. We use AkashAI