It is intuitively easy to understand why self play *can* work for LLMs, if we are able to provide a value function at intermediate steps (although not as clearly guaranteed as in two-player zero-sum games).
In chess / go / poker, we have a reward associated with every next move, but as Noam points out, natural language is messy. It is hard to define a value function at intermediate steps like tokens. As a result, in usual reinforcement learning (like RLVR), LLMs get a reward at the end. They end up learning to 'meander' more for hard problems. In a way, we reward brute forcing with more tokens to end up at the right answer as the right approach.
However, at
@DeepCogito, we provide a signal for the thinking process itself. Conceptually, you can imagine this as post-hoc assigning a reward to better search trajectories. This teaches the model to develop a stronger intuition for 'how to search' while reasoning.
In practice, the model ends up with significantly shorter reasoning chains for harder problems in a reasoning mode. Somewhat surprisingly, it also ends up being better in a non-thinking mode. One way to think about it is that since the model knows how to search better, it 'picks' the most likely trajectory better in the non-thinking mode.