have been recently thinking about why pretrain research matters among the seemingly more crucial data/compute/rl bottlenecks and sharing my take here on what makes pretrain research (still!) vital: 1. better computational efficiency: scalinglaw shifts, 2x less FLOPS needed to achieve the same loss, etc. plus e.g. long context settings where switching to hybrid or sparse attn can save you >90% FLOPS. many model arch / optimizer improvements can save you >20% flops needed for the same loss - those are research innovations on every axis from training iter dimension to inter-layer and intra-layer. the effect of compounded architecture advantage is very distinctive given that ur always improving against your sota baseline. good pretrain research might very well have already delivered you a 10x more efficient (and likewise, better under the same compute) model arch compared to three years ago, and there's still obv many inefficiencies left to be optimized. over half of the compute is still spent on pretraining when you do new from-scratch model trainings rn, and having weeks & months saved there could really allow much more rapid iterations across the entire stack, compounded. 2. to train models one couldn't have been able to previously: residuals, optimizers, etc. this one's less common since most of the arch innovations don't offer more beyond the expressivity gain. but there are significant ones which can e.g. provide more stable learning dynamics (both theoretically and in practice) at all scales so one could scale up. new model configs or forms of training also come back to better efficiency data/compute/FLOPS bottlenecks certainly exist but are relatively more orthogonal to pretrain research and imo it is unclear whether one will be a clear intelligence bottleneck a year from now than the other. in hindsight ive been using "pretrain research" tho this itself is an inefficiency (with further inefficiencies under its scaling law) and "deep learning research" is a better phrasing.

Reinforcement learning has exploded on Modal, and we've been cooking. Here's a review of lessons learned helping teams train at scale, the patterns we kept seeing, and an open-source library to get started with RL on Modal quickly.

mathematicians love putting "an introduction to" in the title of a textbook that contains the most fucked up and domain specific information possible

This woman just shattered everything I thought I knew about quantum physics in 60 seconds 😱 "Quantum physics isn’t just science, it’s logical spirituality. You don’t attract what you want. You align with what you already are." Every version of you already exists in the quantum field - The wealthy you, the happy you, the successful you. "The quantum field doesn’t respond to begging. It responds to certainty." Your most powerful tool is visualization. ‘Our minds don’t know the difference between imagination and reality.’ Time isn’t linear, you can pull your future into the now & re-code your past. You are not a person inside the universe. You ARE the universe experiencing itself through you. "Go live like the miracle you are.”

With great {malloc} power comes great responsibility

Liftoff of Starship on its twelfth flight test

FWIW, I think this moves up my AI timelines a bit. I think the next milestone will be "Artificial *Grothendieck* Intelligence" (AGrI): defining new general mathematical structures to solve the hardest of open problems as special cases, like the Riemann Hypothesis or P vs. NP. What impressed me about the OpenAI planar unit-distance result is not just that it solved a hard problem, but the particular way it seems to have done so. For decades, the expert intuition was that the best constructions should look roughly grid-like. That intuition was *not* obviously silly; it was held by extremely serious mathematicians (of the likes of Erdos!). And yet the model found a new family of constructions that defeated it, based on literature in other areas of mathematics. This feels like one of those cases where the "vague idea" is natural, but the solution lives in a huge space of possible design choices: which symmetries to preserve, which to break, which parameters to introduce, which ugly cases to try, which seemingly-unmotivated configurations to keep exploring. Humans tend to navigate that space with aesthetic priors. We get embarrassed by ugly constructions. We avoid paths that do not look conceptually clean early on. The model seems much more willing to "fearlessly" plough through the design space until something works. I imagine a lot of open problems in mathematics (and theoretical computer science!) may have a similar flavor, and would not be surprised if many of them start to fall soon. But for the "very big" problems, maybe extensive search through constructions in the vast existing literature is not enough. Maybe what is needed for those problems is closer to Grothendieck-style mathematics: inventing the right ambient language in which the original problem becomes a special case of a more general structure. That's what I mean by Artificial Grothendieck Intelligence (AGrI). Not merely AI that proves theorems, but AI that invents the new mathematical objects in which the theorems become *inevitable*. And why stop at one AGrI? You could imagine simulating something like the IHES school: manager agents dividing a research program into subprograms, subagents pursuing lemmas for hours or days, other agents distilling the resulting abstractions, checking them, and communicating the useful pieces back upward. One reason Grothendieck's IHES school was so successful is that its abstractions were relatively human-compressible. Once you adopted the relative perspective, the ideas could propagate through the community. But maybe that constraint has also been a bottleneck. Maybe many longstanding open problems, like those in number theory which Grothendieck felt was the hardest nut to crack, have solutions that are checkable in principle, but whose motivating abstractions are not human-compressible. In fact, I would wager that many, if not all, of these longstanding, open human conjectures live in PSPACE, but PSPACE is massive! I could imagine the AGrIs of the future might easily find non-human compressible abstractions that can be checked in PSPACE, but are infeasible for any human to check manually. Thus, the next frontier may be mathematics that is machine-discovered, machine-compressible, and machine-checkable — beautiful, in a different way to the machines, but not necessarily in the human way. I can't wait to see what open problems get solved next. What an exciting time to be alive.

Today, we share a breakthrough on the planar unit distance problem, a famous open question first posed by Paul Erdős in 1946. For nearly 80 years, mathematicians believed the best possible solutions looked roughly like square grids. An OpenAI model has now disproved that belief, discovering an entirely new family of constructions that performs better. This marks the first time AI has autonomously solved a prominent open problem central to a field of mathematics.

this is the foundational infrastructure for multiplanetary civilization

Palantir embraces the neurodivergent. Join us.

The Manhattan Project 2.0 - the Secrecy of UFO Crash Retrieval Programs youtube.com/watch?v=jR-h5p2b… Part 1 of 2 into investigating the security architecture behind UFO crash retrieval and reverse engineering programs. This massive exploration seeks to uncover the who, what, when, where, and why behind the covert nature and security architecture of the UFO Legacy Program portfolio. Best visualized as an onion, the UFO portfolio secrecy apparatus is seemingly structured as an infinite series of "layers", where the center of the onion is the core truths, materials, senior leadership, biologics, etc of the behind the UFO effort and seemingly infinite layers of protectoin exist to keep even presidential administrations and "temporary employees" one layer away from the beating heart of UFO Legacy Programs. Today's project seeks to explore the very formation of this onion.... the "Manhattan Project 2.0." Indeed, today, the UFO Legacy Program portfolio seemingly operates in an extremely compartmentalized structure, featuring a fractured control group and select few personnel who can peer across the "silos" into other programs housed under a bizarre quasi-government and industry control group.... But this wasn't always the case.... As this project will explore, a centralized UFO recovery and exploitation effort was founded in 1947, and the security apparatus was ripped straight from the Atomic Project and its predecessor, the Atomic Energy Commission. The Manhattan Project 2.0 directly translated several critical secrecy concepts from the Manhattan Project to make this UFO crash retrieval effort stalwart and impenetrable, including: compartmentalization, organizational architecture, physical security, classification/info control, and political shield/cover. This project will seek to explore how Presidents Eisenhower and Truman emboldened the early legacy effort by vesting custody of NHI disks within the Atomic Energy Commission National Labs, establishing the UFO control group under the National Security Council and NSC's 5412 Committee, and safeguarding the programs with statutory authority under the 1954 Atomic Energy Act. Today will also explore key architects in the Manhattan Project 2.0s security infrastructure, including Dr. Vannevar Bush, who also essentially established the modern-day scientific military structure, and General George C. Marshall who seemingly outlined early UFO Crash Retrieval teams through the Alsos Missions and T-Force. Finally, today will explore how the Cold War, specifically the 1980s, broke a once unified Manhattan Project 2.0 into the modern-day siloed and shattered program infrastructure. Specifically, a thorough analysis will focus on presidential executive orders expanding or constraining special access program oversight, the Yellow Fruit and Iran-Contra scandals, and finally 1980s Pentagon audits that, as the video argues, nearly exposed the legacy programs. Short list of the MANY things we will explore today: - Translation of compartmentalization, organizational architecture, physical security, classification/info control, and political shield/cover from the Manhattan Project onto the Manhattan Project 2.0 - Establishing of the Manhattan Project 2.0 during 1947-1951 - President Truman establishing the UFO Legacy Program control group within the National Security Council - Eisenhower taking the above point a step further by instilling the UFO legacy program control group within the National Security Council 5412 Committee "Special Group" which would then continue into Nixon's 303 Committee - Truman vesting custody of recovered discs within the Atomic Energy Commission's National Laboratories in 1948 to undergo scientific analysis by teams led by Dr. Vannevar Bush - Dr. Vannevar Bush and General George C. Marshall as several of the primary architects behind UFO legacy program secrecy, scientific analysis infrastructure, and crash retrieval teams - Analysis of changing US classification systems and introduction of "ad hoc" special access controls onto "black programs" with Eisenhower's Executive Order 10501, which removed the "restricted" classification from standard executive order-based confidential, secret, and top secret classification - Study of the 1954 Atomic Energy Act's statuatory authority being implemented to subject the Manhattan Project 2.0 to even greater security controls and secrecy - Explanation of "restricted data" and how AEC/DOE/NNSA classification systems vary drastically from standard US executive order-based classification - Breaking of the Manhattan Project 2.0 into the siloed "rice bowl" of programs which exist today during the Cold War - Study of Special Access Program oversight flexing or tightening during the 1950s-1980s through several executive orders, including EO 10501, EO 11652, and EO 12356 (and NSDD-159) - Investigate 1983's audit of the "Yellow Fruit" USAP out of the Army Special Operations Division, leading to MASSIVE SAP reforms, and why I believe such audits that helped lead to the Iran-Contra Scandal also exposed the UFO legacy program portfolio - Explore how after this fracturing, finally in 1994, the program structure broke itself into a quasi USG/industry group of solely several dozen top level individuals Discuss instances of Cover Programs (USAF Outside Activities) and SAPs hidden within Cover Programs (Navy Sand Dollar) - So much more! Part 2 look for: - Modern-day onion structure - full overview and explanation of Special Access Programs - Legacy program protection offices - "unrivaled secrecy" - Disinformation / narrative management Thanks for watching, see you on part 2!

When DeepSeek Code?