Thank you, Mamma, for the birthday wishes! ❤️

Congrats to all the Knicks fans!! including my dad @Cornellian1988! :)

OpenEvidence is understandably not happy with the recent LLM benchmarking study! I agree with the larger point of better benchmarks needed. Perhaps OpenEvidence could be evaluated on our open and completely transparent Medmarks benchmark suite 🙂

At Stability AI, I contributed to HDiT (invented by my colleague @RiversHaveWings) as a side project to explore some diffusion research outside of the medical AI work I was doing. What's interesting to see is how this project has also been used in medical AI research as well! In this paper, the authors use HDiT directly applied to the wavelet domain to train a diffusion model to generate 3D brain MRI scans.

This study has been going viral. I think that most people are misunderstanding its conclusions a bit. This paper DOES NOT MEAN domain-specific models are not worth it. First of all, UpToDate and OpenEvidence are not models, but products. And there's no information on what models they are built on. They are likely built on top of older models. For all we know they're built on top of gpt-4o or Llama-3.1 or something 🤣 (it's probably something more recent/powerful than that but just trying to emphasize the point) Second of all, the benchmarks are a bit limited. The benchmarks include MedQA (which is pretty saturated at this point), HealthBench (which focuses on patient conversations), and a closed dataset of doctor questions to an LLM. There are many aspects of clinical use of LLMs that are not at all analyze by this benchmarking approach. What conclusions can be made then? Only that UpToDate and OpenEvidence is worse than frontier models on the limited set of benchmarks tested in this paper. It doesn't mean that domain-specific models cannot beat general purpose models. In fact, we have done a comprehensive benchmark (medmarks.ai) which includes MedQA and HealthBench, among many other benchmarks. We look at general-purpose models, and versions of those same models but adapted for medicine. There seems to be a noticeable boost going from general-purpose to medical fine-tune. So if you took a frontier model and were able to fine-tune for medical applications it would definitely be better. i.e. a domain-specific model would be better. It is true that the current domain-specific models (which are often built on open-source models that are not at the frontier) are often worse than frontier models. It is not true that building domain-specific models cannot beat general-purpose models. I think the main problem is open-source models aren't progressing fast enough with respect to the frontier models and on top of that very few groups are adapting them quickly enough to release better and better medical AI tools. Some groups claim to have medical-specific models that outperform frontier models, ex: Baichuan-M4. Hopefully we'll see more medical-specific models trained on top of really strong base models come out soon.

@iScienceLuvr happy birthday my friend!!! ❤️

Thank you, Parm, for the birthday wishes! 🙏

23 yrs old... so I guess I'm an unc now, right?

Thank you to my co-founder and close friend @humanscotti for the birthday wishes!

Cheers to 23 years of awesomeness 🎉🎂 Blessed to have you in our lives ❣️ Wishing you a year filled with blessings, exciting discoveries, and continued impact on the🌎

Has anyone confirmed if fable is degraded solely on frontier llms or all frontier models? Does it need to be autoregressive?

Congrats to @stevejang and @km! We are lucky to have them as investors in @SophontAI. They have been very helpful in this early stage of our startup, glad that Kindred believed in our vision very early on.

Claude Fable 5 is likely very capable inherently on healthcare. That's great! Too bad it's near impossible to tap into those capabilities due to their extremely sensitive safety filters. I hope this is adjusted going forward.

"Sometimes you just kinda phone it in for a year, you know?" The YOLO v3 paper is a complete banger imo

DO NOT USE FABLE 5 FOR AI R&D/CODING!!

I appreciate Anthropic has provided healthcare-related evals! Let's quickly go over them. HealthBench - benchmark from OpenAI that includes 5k multi-turn conversations with patients, and rubrics for evaluation. Fable 5 achieves 62.7% vs. GPT-5.5's 56.5%. Personally, I would also appreciate HealthBench Hard scores as well in addition to the aggregate score. HealthBench Professional - another OpenAI benchmark that focuses on physician tasks. Fable 5 achieves 66.0% vs. 51.8% for GPT 5.5. HealthAdminBench - a computer-use benchmark from Stanford that evaluates the completion og various administrative tasks (prior auth, denials/appeals, etc.). Fable 5 achieves 51.9%, no GPT-5.5 score provided. Overall, this models seems to perform quite well on healthcare benchmarks. Would also appreciate additional benchmark scores like MedCalc-Bench (which was previously reported by Anthropic) and MedXpertQA (an unsaturated, hard medical MCQA benchmark). Glad to see frontier labs are more comprehensively benchmarking and reporting the medical capabilities of their models!