804 Photos and videos
Building an LLM means evaluating it over & over as it changes. Tweak a hyperparameter or scale the model up, & every new checkpoint sends you back through the same benchmarking loop.
We're releasing olmo-eval, a workbench built for this kind of iterative model development. ๐งต
2
11
70
3,685
If you find yourself asking "how does this model checkpoint differ from the last, and where did it improve/regress?", that's what olmo-eval is for.
We're releasing it openly so the community can build on it.
๐ป Code: github.com/allenai/olmo-eval
๐ Blog: allenai.org/blog/olmo-eval
1
7
732
Ai2 retweeted
So impactful! Excellent work from @sewon__min et al! Alternatively, it points to so much about โnoveltyโ of a generation and trace its history: clearly critical for scientific discovery with these models!
LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work.
So what is a modern LLM built on? Olmo 3 โ 89 model 183 dataset dependencies; Nemotron 3 โ 273 560
We made ModSleuth to trace this. ๐งต
1
14
2,743
Ai2 retweeted
Jun 11
One day I tried tracing all of Olmo's dependencies manually. A few hours later, I realized I can't do it and gave up. Then @sadhikesaven and @CoderBak ModSleuth ๐ฅ
Turns out Olmo and Nemotron have hundreds of dependencies that are super deep, recursive, and not easily visible. I'm glad I gave up early ๐
Spoiler: I thought this would be a one-week Claude Code project. It was not.
The hard part wasn't information extraction (which Claude Code is good at). The hard part was something much trickier. Check out the paper to learn more!
(And yes, if a model release says it used Claude Code, ModSleuth will trace that too... which means the model depends on Claude Code, which has its own dependencies, and ModSleuth itself depends on Claude Code ๐คฏ)
LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work.
So what is a modern LLM built on? Olmo 3 โ 89 model 183 dataset dependencies; Nemotron 3 โ 273 560
We made ModSleuth to trace this. ๐งต
2
21
145
22,251
LLMs are no longer created w/ human data alone. They rely on other models to generate & filter data, evaluate outputs, & guide dev work.
So what is a modern LLM built on? Olmo 3 โ 89 model 183 dataset dependencies; Nemotron 3 โ 273 560
We made ModSleuth to trace this. ๐งต
5
41
250
85,450
As LLM pipelines become more complex, we need tools like ModSleuth to find out & identify what artifacts models are built on.
โถ๏ธ Demo: modsleuth.cal-data-audit.org
๐ Paper: arxiv.org/abs/2606.12385
1
17
1,376
Ai2 retweeted
Jun 10
๐๐๐๐ฎ๐ฆ-๐ฆ๐๐ถ๐๐๐ , our new climate emulator that learns to separate the effects of sea surface temperature & CO2, is now on @huggingfaceโcheck it out โ huggingface.co/allenai/ACE2Sโฆ
Today we're introducing ACE2S-SHiELD , a climate emulator that learns to separate the effects of sea surface temperature & CO2.
It accurately handles scenarios where previous versions of our ACE family of climate emulators produced inaccurate results. ๐งต
2
4
2,631
Ai2 retweeted
Jun 10
I'm hiring Senior Research Engineers. Come build open-source vision-language models from zero to hero: pretraining, mid-training, post-training, RL, the whole pipeline.
job-boards.greenhouse.io/theโฆ
2
14
168
21,515
This work was done in collaboration with @NOAA's Geophysical Fluid Dynamics Laboratory.
โ Read more about ACE2S-SHiELD in our preprint: arxiv.org/abs/2606.07928
3
641