- Tweets 2,042
- Following 1,141
- Followers 2,796
- Likes 7,389
ALT AI@HPI Conference logo
ALT Token Distillation: Attention-Aware Input Embeddings for New Tokens Abstract: Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines.
ALT panelists on stage
ALT panelists on stage
ALT Stephen Casper (MIT) on stage speaking
ALT Anthony Aguirre on stage at UNESCO
ALT Gillian Hadfield on stage with Yoshua Bengio and Susan Leavy
ALT University campus at lake
ALT 2 reviewers raised their scores but โWe will be reverting reviews and scores to their state before the discussion perio
ALT Signed Declaration for European Digital Sovereignty (18 November 2025)
ALT Gerard de Melo at AI@HPI Conference
ALT Token Distillation: Attention-Aware Input Embeddings for New Tokens Abstract: Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines.
ALT Gerard de Melo during a video shoot for a social media reel