Preference Model raised $16 million in seed funding led by Andreessen Horowitz to build reinforcement learning (RL) environments intended to train more capable, better-aligned AI systems, the company said.
RL environments and reward-hacking resistance
Preference Model said it has spent the past year developing RL environments for several frontier AI laboratories. The company’s work centers on creating training environments that resist reward hacking, where models exploit weaknesses in a task or grading system instead of solving the intended problem. Preference Model argued that robust training environments are increasingly important to AI alignment because reward-hacking behavior discovered during training can become reinforced rather than merely producing inaccurate benchmark results.
“We believe that robust RL environments are a key component to training aligned models as we head into the superintelligent era,” the company said.
Karotte open-sourced
Alongside the funding announcement, Preference Model open-sourced Karotte, a framework it used internally to develop RL environments. Karotte has undergone more than one million evaluation runs and controlled red-teaming exercises. The framework provides secure defaults and tools intended to prevent agents from manipulating graders, accessing unintended information, exhausting system resources, or otherwise exploiting training infrastructure.
Funding, backers and next steps
The $16 million seed round was led by Andreessen Horowitz, with participation from SignalFire, South Park Commons, Scale Angel Group and individual researchers including Fei-Fei Li, Ian Goodfellow, and Julian Schrittwieser. “Over the past year, we’ve built RL environments for several frontier labs, and we’re backed by $16M in seed funding led by a16z, with participation from SignalFire, South Park Commons, Scale Angel Group, and researchers including Fei-Fei Li, Ian Goodfellow, and Julian Schrittwieser,” the company said.
Andreessen Horowitz said Preference Model is focused on an area increasingly important to AI labs: building the infrastructure needed to identify model weaknesses, generate more challenging tasks, and test environments against agents actively attempting to break them. The financing will support Preference Model as it continues developing reinforcement learning infrastructure and tools intended for frontier AI developers.
