Today, we're joined by @Muennighoff, a PhD student at @Stanford University, to discuss his paper, “S1: Simple Test-Time Scaling.” We explore the motivations behind S1, as well as how it compares to OpenAI's O1 and DeepSeek's R1 models. We dig into the different approaches to test-time scaling, including parallel and sequential scaling, as well as S1’s data curation process, its training recipe, and its use of model distillation from Google Gemini and DeepSeek R1. We explore the novel "budget forcing" technique developed in the paper, allowing it to think longer for harder problems and optimize test-time compute for better performance. Additionally, we cover the evaluation benchmarks used, the comparison between supervised fine-tuning and reinforcement learning, and similar projects like the Hugging Face Open R1 project. Finally, we discuss the open-sourcing of S1 and its future directions. 🎧 / 🎥 Listen or watch the full episode on our page: twimlai.com/go/721. 📖 CHAPTERS =============================== 00:00 - Introduction 1:56 - S1 and o1 models 2:42 - Approaches to test time scaling 6:45 - Comparison of S1 and R1 models with o1 model 9:19 - Dataset curation 16:53 - Metrics 18:14 - Budget forcing 23:51 - “Wait” insertion 29:06 - Decontaminating samples in datasets 30:12 - Rejection sampling 32:05 - Open-sourcing S1 33:03 - Other model families 35:20 - Biases in model families 35:49 - Evaluation 36:56 - RL versus SFT 39:12 - RL in R1 40:04 - RL in training recipe 46:12 - Future directions

DeepSeek r1 is exciting but misses OpenAI’s test-time scaling plot and needs lots of data. We introduce s1 reproducing o1-preview scaling & performance with just 1K samples & a simple test-time intervention. 📜arxiv.org/abs/2501.19393

详细内容：xiaohu.ai/c/ai/s1-simple-tes… 论文：arxiv.org/pdf/2501.19393

s1 - a new simple! open-source test-time scaling approach from @Stanford. With s1 researchers found the simplest way to improve reasoning through test-time scaling. s1's innovations: • A small s1K dataset with 1,000 tough and diverse questions, each with detailed reasoning steps for training. • Budget forcing to controls how long the model thinks. What about the results of s1? - It gets up to 27% higher scores on math problems compared to OpenAI’s o1-preview model. - Performance jumped from 50% to 57% on a math competition even without extra test-time optimizations. More details:

s1: Simple test-time scaling "We show that training on only 1,000 samples with next-token prediction and controlling thinking duration via a simple test-time technique we refer to as budget forcing leads to a strong reasoning model that scales in performance with more test-time compute."

Following up on my reasoning model article, I just read the new "s1: Simple Test-Time Scaling" paper, which describes an interesting method for improving reasoning models using a combination of pure supervised finetuning (SFT) and scaling inference compute. In short, their approach is 2fold: 1. Create a curated SFT dataset with 1k examples that include reasoning traces. 2. Control the length of responses by: a) Appending "Wait" tokens in certain cases to get the LLM to generate longer responses, self-verify, and correct itself, or b) Stopping generation by adding an end-of-thinking token delimiter (“Final Answer:”). They call this length control "budget forcing." Budget forcing can be seen as a sequential inference scaling technique because it still generates one token at a time (but just more of it). In contrast, we have parallel techniques like majority voting, which aggregate multiple independent completions. They found their budget-forcing method to be more effective than other inference-scaling techniques I’ve talked about, like majority voting. If there's something to criticize or improve: I would’ve liked to see results for more sophisticated parallel inference-scaling methods, like beam search, lookahead search, or the best compute-optimal search described in Google’s "Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters" paper last year. Or, even a simple comparison with a classic sequential method like chain-of-thought prompting ("Think step by step"). Anyway, it’s a really interesting paper and approach! Bonus: Why "Wait" tokens? My guess is they were inspired by the "Aha moment" figure in the DeepSeek-R1 paper, where researchers saw LLMs coming up with something like "Wait, wait. Wait. That’s an aha moment I can flag here." which showed that pure reinforcement learning can induce reasoning behavior in LLMs. Interestingly, they also tried other tokens like "Hmm" but found that "Wait" performed slightly better.

DeepSeek r1 is exciting but misses OpenAI’s test-time scaling plot and needs lots of data. We introduce s1 reproducing o1-preview scaling & performance with just 1K samples & a simple test-time intervention. 📜arxiv.org/abs/2501.19393