personally I feel like the inflection point was early 2022. The sweet spot where clip-guided diffusion was just taking off, forcing unconditional models to be conditional through strange patchwork of CLIP evaluating slices of the canvas at a time. It was like improv, always trying to riff of mistakes and sitting right at the fine line between interesting and incoherent.

Honestly I’d expect Shampoo to beat Muon, shampoo with b2=0.0 and momentum covariance (instead of grad), whitening only on one side should be equivalent to muon up to numerical imprecision. Question is whether it’s worth it, it’s tracking 4 states at 2 x n^2 x m^2 memory and while NS can be used for the power raising instead of eigh, still more flops. In single device training it’s a good consideration, making it work well in distributed feels like a nightmare, I’d be more inclined to give SOAP the chance if paying that cost, Adam in the eigenbasis is a genuinely unique effect, but yeah there’s a lot of engineering obstacles that even in best case make it a hard sell

fella said an exact claudism is Claude built in the image of Dario or did he become like Claude?

Different from EMA of weights compared to gradient, but AdEMAmix works gorgeously for diffusion and I suspect there’s a similar thing going on there

The alignment problem in humans and institutions feels measurably more concerning than it does in AI right now

Feels like this post was the wake-up call for many, but felt evident from the hyper-optimized cuda kernels beyond what humans write, Karpathy's auto research and more. Notably, these are all open-source projects. You can extrapolate whats happening behind the scenes from there

REPA thoughts REPA as x0 prediction - the DINO/Encoder features prediction can be seen loosely as "x0 prediction in an alternate representation space than main diffusion space, from an intermediate/rather than final hidden state" - If so, we should consider the same principles for predicting x0, like loss weighting etc. - simply: we know noise prediction is "impossible" from fully clean image (we know nothing about the noise sample), x0/clean image prediction is "impossible" from fully noised state (we know nothing of the image). velocity as (x0 - eps) suffers at both endpoints, at x0 we know nothing of eps and can't really predict this direction, and vice versa - the general form of this is both endpoints of the trajectory are a domain, and when we're on the opposite side, the minimum risk prediction is to predict the mean of the opposing domain/dataset (or conditional slice if relevant) - Because of this, predicting DINO features at a fully noised state may not give us a helpful signal yet might be disproportionately weighted due to high error/grad magnitude. IREPA claim that structure > representation quality - If so, representation space features might be superfluous - "Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model" demonstrates that image diffusion models naturally develop a representation of depth, fairly reliably predicted even at high noise levels with only a linear probe - This observation could be "flipped on its head" we know a performant pretrained model has this quality in the end, we could do the inverse and add supervision/guidance to encourage this early - segmentation maps, depth, etc feel like good candidates if its spatial/structure properties that drive the improvement as they nicely correlate to low frequency coarse structure Mapper design, spatial target parameterization axes - a few papers have covered this, but the design of the mapper and level of compression describes the assumption we're making and guidance we enforce - Spatial/Compression: i.e. full spatial patch-to-patch matching, pooled, or something in between like a downsample/perceiver pooling - Depth and Architecture of mapper: how much of the burden of adaptation for this secondary task falls on the backbone model vs the actual mapper parameters. - are we wanting to be a linear projection away from the reference space? a shallow MLP away? a deeper mapping? -- My impression is the shallower options deliberately ask the diffusion representation space to already be somewhat denoised (because we're mapping to clean x0 in DINO space) and a more trivial mapping away. -- Deeper options can sort out the noise filtering and more extensive work to map to DINO space, if that's adapted/isolated to the mapper, then i feel the question is less about taking on a shape that resembles the reference feature space, and instead having high mutual information. To successfully predict the features, the information has to be there, any variables around getting to that other space the mapper squares away.

Coordinates were not meant to be discrete, you risk class imbalance and no intrinsic knowledge of number line or a loss that caters to that, just use a linear head to output 4 coordinates with an MSE loss. Fourier embed at input side. Here’s a paper we did on that

i think we should be talking more about the Taalas 16k tok/s chips. If this scales, its a good partial counter to the AI bubble. Loss leaders need to hold off long enough for the 10-100x inference cost reductions.