Testing LLMs (and prompts) like we test software: towardsdatascience.com/testi… TL;DR: (1) You should, (2) How to test: specific properties, evaluate these with LLMs (perception is easier than generation), (3) What to test: get the LLM to help you figure it out.

Microsoft open-sources a new AI library that connects to open-source GPTs, not just OpenAI. github.com/microsoft/guidanc…

Also highly relevant: guidance from microsoft "Guidance programs allow you to interleave generation, prompting, and logical control" Also internally handles subtle but important tokenization-related issues, e.g. "token healing". github.com/microsoft/guidanc…

been reading the readme for github.com/microsoft/guidanc…, kind of galaxy brain tl;dr they made a whole prompt engineering language

Blog post: playing with Vicuna-13B, ChatGPT (3.5), MPT-7B-Chat on harder stuff medium.com/@marcotcr/explori… TL;DR: We think ChatGPT is still way ahead, but sometimes the extra control from open source models is worth it.

My intern is close to writing a paper, so I wrote her this blog post on writing (part 1 of 2): medium.com/@marcotcr/writing…

I never tweet, but here is a blog post I wrote for an intern, may be useful for others too... Part 1: medium.com/@marcotcr/coming-… Part 2: medium.com/@marcotcr/organiz…

"Why Should I Trust You?" Explaining the Predictions of Any Classifier. Promo video: youtube.com/watch?v=hUnRCxny… #kdd2016 @guestrin @sameer_

Great work from @marcotcr , @sameer_ on explaining any machine learning model (20 newsgroup, deep net). homes.cs.washington.edu/~mar… @guestrin

oi