day 2 of closed discussions on agi and asi. today's debate was whether ai companies should build superintelligence. i was on team b, the against side, with an oxford philosopher and an oceanographer turned ai safety writer. team a was a diplomat, a cambridge phd scholar who is also hod of computational sciences, and a formal verification engineer.
their case was rhetorically strong. 10M cancer deaths, 15M tuberculosis, 7M air pollution. not building was framed as abdication.
we defended with 3 pillars. 1, anthropic's own alignment faking research shows 12 to 78 percent compliance shift on monitoring framing alone. 2, in my own research sprint, 1 learning rate change moved a null result to 51x behavioural drift. 3, anthropic shelved mythos on 7 april. it leaked through a contractor breach within a week. half life from too dangerous to release to broad access is days, not quarters.
their governance counter was a list of treaties and institutes. voluntary, early stage, untested by their own admission. evidence people are worried, not evidence the problem is solved.
team b won on substance. facts that go uncountered are the room hearing you.
pause is not never. pause is build the verification infrastructure first.