Discriminative Protein Sequence Modelling with Latent Space Diffusion @instadeepai 1. This paper introduces a novel Latent Space Diffusion (LSD) framework aimed at improving protein sequence representation learning. The approach combines manifold learning via an autoencoder and distributional modeling using a denoising diffusion model applied to the learned latent space. 2. The LSD framework proposes two architectures: LSD-TN and LSD-NM. LSD-TN employs a homogeneous model where amino acids of the same type are identically distributed in the latent space, enhancing robustness. LSD-NM uses a noise-based variant of masking that applies varying levels of corruption to improve generalization. 3. Unlike conventional masked language models (MLMs), the LSD framework replaces masking with Gaussian noise to produce continuous, structured representations. This allows the model to capture long-range dependencies more effectively. 4. The diffusion model is trained on latent embeddings obtained from the autoencoder, which improves discriminative performance across a variety of protein prediction tasks, including thermostability, human-protein interactions, metal ion binding, and subcellular localization. 5. The model evaluation shows that diffusion representations trained on LSD models (LSD-TN and LSD-NM) outperform those trained on MLM baselines. The LSD-NM model achieves particularly strong performance in predicting human-protein interactions, suggesting complementarity between the two architectures. 6. The study introduces a Token Norm bottleneck for the LSD-TN model, which partitions the latent space embeddings by amino acid type, enhancing interpretability and model robustness. This design choice is particularly effective in improving classification tasks. 7. Noise Masking in LSD-NM is designed to enhance the diffusion model’s performance by varying the corruption level applied during training, making the model more robust to noise and improving its discriminative capability. 8. Evaluations indicate that while the LSD framework shows promising results, the MLM encoder representations still outperform the diffusion representations. This suggests that further architectural improvements are needed to match or exceed traditional MLM-based models. 9. Future work aims to enhance the generative capabilities of the LSD framework, optimize the latent space, and investigate hybrid approaches that combine MLM and diffusion-based methods for improved representation learning. 📜Paper: arxiv.org/abs/2503.18551 #LatentSpaceDiffusion #ProteinModelling #MachineLearning #Bioinformatics #Autoencoders #DiffusionModels #ProteinSequenceLearning #DeepLearning