we're launching 🤗 yourbench today, an open source tool for custom benchmarking and synthetic data generation from ANY of your documents. it's a big step towards improving how model evaluations work early access link in replies! (1/8)

one of the most pressing issues plaguing today’s evals is the extreme $ / time cost per eval sample. Needing an entire agent rollout seems extremely slow, especially when you’re trying to use said eval as a signal for posttraining or dev splits for GEPA style setups (cc: @lateinteraction) I don’t know if there’s a solution to this, but it seems desperately needed I like to think of it from the perspective of human interviews. Interviewing for L9 doesn’t take exponentially more time than L3. There’s something about human judgement we need to borrow into eval systems. Have we ever explored adversarially adaptive evals well?

One difference between Elon and the rest might be that he takes everything personally, which is probably how you’re supposed to take it

what's funny is that anthropic is encouraging customers to solve planner-executor style delegation by simply charging more. class act

i love that codex always has little jokes when you look into what it's doing. @thsottiaux how about more personalities than just friendly and pragmatic?

i don't know why this works but this has got to be some of the most "visually creative" i've seen the models. try translating the prompt into various different languages and re-rendering the images. freakish

"llms are a trick" he says, as he makes the same content for the 85th time, much like a next token predictor

We compare the distributions of real and simulated user behaviors. A few takeaways from our results across 24 LLMs: - Scale alone isn't enough: Llama-3.1-8B-Instruct beats Llama-3.3-70B-Instruct - 8B models specifically trained as user simulators rival the best closed-source models - Open-source is competitive: gemma-4-31B-it and gpt-oss-120b outperform several closed-source models

What happens when you compare the distributions of real and simulated user behaviors? 🔍 The gap is large. We introduce a method to measure this gap and evaluate 24 LLM-based user simulators across coding and writing tasks. @convai_uiuc @MSFTResearch @berkeley_ai 🧵 1/N

A. they do some kind of vector steering towards exploration that just boosts "goblin" related logits? B. pre-training / post-training quirk like the em-dash? I think its B, but A would be monstrously cool

What are we even doing here man

What we really need to do is communicate our desires better to the models (the issue) because the cost of implementation tends to 0 (the actual PR) Very strange times ahead because we’ve been training engineers to do more of the latter and not enough of the former

Feelings drive reasoning. Emotions drive reasoning. Being curious allows you to explore a file system. Being anxious lets you write better test cases. By this logic humans are lumps of meat with electrochemical signals and don’t need any of these either.

Opus 4.7 is a significantly smaller model (probably ~4.6 Sonnet scale) distilled from Mythos Why? Mythos has been internally avail since early feb giving them ample time to pretrain and logit distill from mythos a small focused base also addresses their compute crunch issue

If you've tried out openclaw in the past, and found it too sloppy, I'd highly encourage you all to give hermes agent by @NousResearch a try. Give it access to emails (with a local model if you're scared), and keep an open mind. :)

pro tip: scale is all u need, actually

What would be interesting for anthropic to do here is public ablations on their own models! What about opus with haiku? How much better or worse is that? What about sonnet with haiku? Anyone have papers that investigate this well that you recommend?

academia is so broken man this is why as a reviewer you see papers from 2023 using llama-2 7b as a baseline that keep getting recycled hoping for a Hail Mary there HAS to be a consequence to the number of papers authors submit

Lisan is right here. There is no incentive powerful enough to fund the training of a 10T model. China and alibaba are closing down. Reflection et. Al doesn’t have the money. Nvidia doesn’t have the talent.

Can confirm that costs are 10x lesser when you use the prime intellect hosted RL trainer with a cheap Chinese instruct model Qwen3-4B-Instruct-2507 ftw

is it just me or does it make the most sense to interview on weekends (as a candidate)? don't have to move around your regular week to make the times work