Adversarial Sequence Mutations in AlphaFold and ESMFold Reveal Nonphysical Structural Invariance, Confidence Failures, and Concerns for Protein Design 1. A new adversarial study systematically evaluates AlphaFold 3's robustness by introducing point mutations (up to 70%) and deletions (up to 10%) across 200 proteins, revealing striking structural invariance that raises fundamental questions about the model's biophysical reasoning capabilities. 2. The most concerning finding: AlphaFold 3 maintains virtually identical predicted structures even when 40% of residues are mutated with deliberately destabilizing substitutions, or when 10% of residues are deleted—perturbations that would catastrophically destabilize real proteins. 3. This structural invariance persists even for experimentally validated fold-switching proteins, where specific mutations are known to induce alternative conformations. AlphaFold 3 fails to capture these biologically critical transitions, suggesting limited sequence-structure coupling. 4. Confidence metrics prove unreliable: AlphaFold 3's ranking score selects the most accurate structure only ~25% of the time, and these scores correlate more strongly with template availability in the training set than with actual prediction quality. 5. Comparative analysis with ESMFold reveals that the protein language model-based approach shows significantly greater sensitivity to mutations, with structures diverging more rapidly as sequence perturbations increase—suggesting superior learned sequence-structure relationships despite lower absolute accuracy. 6. The study's template analysis provides quantitative evidence that AlphaFold 3's confidence reflects structural similarity to training-set exemplars (Pearson r=0.39) rather than genuine biophysical assessment, indicating heavy reliance on memorized patterns over learned principles. 7. These findings have profound implications for the entire AlphaFold ecosystem: protein design tools like RFdiffusion, binder design methods like BoltzGen and BindCraft, and drug discovery pipelines may inherit these fundamental limitations, potentially generating non-physical sequences or missing viable candidates. 8. The work identifies critical gaps in current structure prediction—models trained primarily on stable, wild-type proteins lack exposure to destabilized mutants and misfolded states, limiting their ability to generalize beyond the training distribution. 📜Paper: biorxiv.org/content/10.64898… #AlphaFold #AlphaFold3 #ProteinStructurePrediction #StructuralBiology #ProteinDesign #MachineLearning #Bioinformatics #ComputationalBiology #AIforScience #ProteinEngineering #DeepLearning #Biophysics

This week we showed near protein-ligand binding prediction SOTA on our lightweight model, Hermes. Unlike other models, Hermes has no way to think about protein or molecule structure. Never seen *any* public data. How did we do it? endpoints.news/leash-bio-war…

An ML drug discovery startup trying really, really hard to not cheat owlposting.com/p/an-ml-drug-… on the 12-person, Utah-based startup @leashbio, their culture of rigor, and the many ways small molecule models accidentally learn the wrong thing

Curious to learn more about the work that went behind making BELKA accessible through Polaris? Learn more about Dataset v2.0 and how we’re helping scientists focus on research, not on data wrangling. 👇🧵 Read the blog: polarishub.io/blog/dataset-v… Access BELKA: polarishub.io/datasets/leash…

We just released 4.25B small molecule-protein measurements - real, physical ones from our lab - on @Polaris_HQ. It's the biggest such public collection ever, please use it! leashbio.substack.com/p/belk…

AlphaFold3 is out with improvements on structural models that include DNA/RNA and small molecules. Unfortunately, there is no code, no binary to run at scale and only a limited webserver. Why even publish? nature.com/articles/s41586-0…