likely latent state and hierarchy around the ar core*. i wouldn't try to ditch that at this point lol. some other interesting bits which make me feel hmm: like the Larimar paper (Larimar: BERT-style encoder writes facts into external memory, whose readout conditions GPT-2 or a GPT-style decoder without weight edits.) and various steering/interp papers using or manipulating reps at various levels to augment the ar core and manipulate the residual stream Larimar is another reminder that memory can be a learned memory interface around the AR core: write/update/forget mechanisms whose readouts condition generation. === >Larimar uses a BERT-style encoder during training and memory writing, but the decoder/base LM is not updated during fact editing. The “memory” gets written/updated, then its readout conditions the decoder. The Larimar paper had three modules: encoder, associative memory, decoder, trained together; then new facts can be added in one shot without retraining/fine-tuning the LLM.