I have mentioned this before - but every industry is now going to have to figure out how to value "task complexity" in the age of AI Healthcare has been doing this for decades with the RVU system, which similarly uses survey methodologies to assign complexity to tasks (and has it's own drawbacks) I'm interested to see how different industries come up with their own methdologies for this

Finally! the first eval ship from cog!!!!!!!!!! 👼🏼 To contextualize: @METR_Evals cap out at ~16 hours. Cog has private enterprise evals up to 100hrs, and is confident enough to put a financial guarantee on it 🤯 METR dataset: ML eng, GPU kernels, cybersecurity > "METR (2026) used a combination of GPT-4o and GPT-5 to estimate the human-equivalent times from compressed Claude Code transcripts. These transcripts were collected from 7 METR technical staff on 34 sessions labeled on human ground truth". rlog​ of 0.83 Cog dataset: real life java/typescript/python/c# feature dev, bugfixes, migrations > "We collected a ground-truth dataset by asking Devin users to review recent representative sessions, and estimate how long each completed session would have taken without Devin. Our dataset consists of 258 sessions from 126 users across a diverse set of enterprise customers." rlog​ of 0.74 on held out set this is pioneering real world evals work and part 1 of a broader frontier code evals drop that I'm really looking forward to writing up. huge kudos to @annarmitchell and @ryanbai1412 for leading the unglamorous last mile data collection!!

LLMs when I ask them for a plan: "this 30-min refactor will take 2 days" LLMs when I give them the full session trace hill climb the prompt: r_log of 0.74 context and prompting is everything

Everyone's talking about AI waste but almost nobody is actually trying to measure the value of AI. The answer isn't blindly cutting spend - it's improving at evaluating what AI is actually delivering. Customers kept asking about this and we wanted to get more rigorous, so we built a methodology to estimate how many productive human engineering hours each Devin session represents. Productive hours are convertible to dollars, which gets you closer to a $ impact for the agent's work. After validating on a dataset of real engineers' time estimates, we're now offering a financial guarantee to back up the value of Devin's work.

AI should earn its keep. Introducing the AI Productivity Guarantee. If Devin delivers less engineering value than you’re paying for, Cognition will fund your usage until it does, up to $10 million. It’s time for the AI industry to stop maximizing tokens and start maximizing productive output.

What’s not shown is the 100 person slack channel coordinating this stunt across offices in 3 continents, to surprise one person. Love this team

Ryan @ryanbai1412 is known for his iconic red hoodie, so for his birthday, everyone at @cognition got their own.

We've raised $1B at a $26B valuation @cognition is firing on all cylinders, and we are constantly building the future of software engineering. Things are moving so fast that you might have missed some of our latest developments: -Having an interface that easily manages agents running in parallel -Choosing the right model for the task, both in accuracy and cost efficiency -Listening to slack or other data streams and automatically triaging -Using auto-fix, bug catcher, and Devin Review to add more layers of security to your SDLC -Having agents automatically fix issues like bugs and vulnerabilities The secret isn't just the models, it's a robust infra platform that enables your org to scale to thousands of agents in the cloud, constantly running and accelerating your roadmap while automatically taking care of tech debt and other issues. It's also a team that gives you outcomes, not dev tools. We work closely with your team on figuring out the biggest problems and solving them together. We genuinely want to help you with your AI transformation in a world where AI form factors change every 3-6 months. I'm so proud of the team here, and grateful to be along for the journey. We have many more announcements to come but we can celebrate today's milestone

What I love about Cognition is that I'm outside of the Silicon Valley bubble. We work with the organizations that the real world runs on - Citi, Mercedes-Benz, the US Army and Navy. As a marketer, telling the stories of how AI software engineering improves the real world, is really motivating.

Reintroducing @Cognition.

The Ruddy Kingfisher. What stunning colours.

Beau Rothrock had been at @AngelList for two months when he walked into a Redshift-to-Snowflake migration in deep trouble, already two months behind schedule. He had a 5-week window to migrate all 14,000 dashboards and reports AngelList runs on. He could've asked for three more engineers and four more months. Instead, he turned to Devin.

the governmental UFO media drop is the best thing to happen to graphic designers since helvetica

Security remediation is an engineering capacity problem. AI has collapsed the time to exploit, but defensive tools haven’t kept up. Today we’re introducing Devin for Security: a set of workflows for reducing security debt, securing every release, and accelerating response

hard

Devining at the SF Symphony (respectfully, during intermission)

Incredibly excited to partner with one of the most technologically innovative companies in history, Mercedes-Benz.

gpt-5.5 unlocks a new level of possibility: