Today, we're excited to announce the acquisition of @velocitytechhq, an AI-powered SRE company founded by @talkain. This move expands the BigPanda leadership team, strengthens our investment in agentic AI & accelerates delivery of AI-driven automation. bit.ly/3JphNk5

Readers responded with both surprise and agreement last week when I wrote that the single biggest predictor of how rapidly a team makes progress building an AI agent lay in their ability to drive a disciplined process for evals (measuring the system’s performance) and error analysis (identifying the causes of errors). It’s tempting to shortcut these processes and to quickly attempt fixes to mistakes rather than slowing down to identify the root causes. But evals and error analysis can lead to much faster progress. In this first of a two-part letter, I’ll share some best practices for finding and addressing issues in agentic systems. Even though error analysis has long been an important part of building supervised learning systems, it is still underappreciated compared to, say, using the latest and buzziest tools. Identifying the root causes of particular kinds of errors might seem “boring,” but it pays off! If you are not yet persuaded that error analysis is important, permit me to point out: - To master a composition on a musical instrument, you don’t only play the same piece from start to end. Instead, you identify where you’re stumbling and practice those parts more. - To be healthy, you don’t just build your diet around the latest nutrition fads. You also ask your doctor about your bloodwork to see if anything is amiss. (I did this last month and am happy to report I’m in good health! 😃) - To improve your sports team’s performance, you don’t just practice trick shots. Instead, you review game films to spot gaps and then address them. To improve your agentic AI system, don’t just stack up the latest buzzy techniques that just went viral on social media (though I find it fun to experiment with buzzy AI techniques as much as the next person!). Instead, use error analysis to figure out where it’s falling short, and focus on that. Before analyzing errors, we first have to decide what is an error. So the first step is to put in evals. I’ll focus on that for the remainder of this letter and discuss error analysis next week. If you are using supervised learning to train a binary classifier, the number of ways the algorithm could make a mistake is limited. It could output 0 instead of 1, or vice versa. There is also a handful of standard metrics like accuracy, precision, recall, F1, ROC, etc. that apply to many problems. So as long as you know the test distribution, evals are relatively straightforward, and much of the work of error analysis lies in identifying what types of input an algorithm fails on, which also leads to data-centric AI techniques for acquiring more data to augment the algorithm in areas where it’s weak. With generative AI, a lot of intuitions from evals and error analysis of supervised learning carry over — history doesn’t repeat itself, but it rhymes — and developers who are already familiar with machine learning and deep learning often adapt to generative AI faster than people who are starting from scratch. But one new challenge is that the space of outputs is much richer, so there are many more ways an algorithm’s output might be wrong. Take the example of automated processing of financial invoices where we use an agentic workflow to populate a financial database with information from received invoices. Will the algorithm incorrectly extract the invoice due date? Or the final amount? Or mistake the payer address for the biller address? Or get the financial currency wrong? Or make the wrong API call so the verification process fails? Because the output space is much larger, the number of failure modes is also much larger. Rather than defining an error metric ahead of time, it is therefore typically more effective to first quickly build a prototype, then manually examine a handful of agent outputs to see where it performs well and where it stumbles. This allows you to focus on building datasets and error metrics — sometimes objective metrics implemented in code, and sometimes subjective metrics using LLM-as-judge — to check the system’s performance in the dimensions you are most concerned about. In supervised learning, we sometimes tune the error metric to better reflect what humans care about. With agentic workflows, I find tuning evals to be even more iterative, with more frequent tweaks to the evals to capture the wider range of things that can go wrong. I discuss this and other best practices in detail in Module 4 of the Agentic AI course on deeplearning.ai that we announced last week. After building evals, you now have a measurement of your system’s performance, which provides a foundation for trying different modifications to your agent, as you can now measure what makes a difference. The next step is then to perform error analysis to pinpoint what changes to focus your development efforts on. I’ll discuss this further next week. [Original text: deeplearning.ai/the-batch/is… ]

A new metric for AI teams: CAIR – “Confidence in AI Results.” In their fantastic article about CAIR, @assaf_elovic and @hwchase17 show how to calculate whether your AI product will succeed or fail based on value/risk/correction (and how the product might change it) blog.langchain.dev/the-hidde… The Hidden Metric That Determines AI Product Success x.com/hwchase17/status/19335… #AI #CAIR

🚀 Today on Velocity Byte Sized: In this walkthrough, we show how you can create a free Postgres database with #CockroachDB and seamlessly connect it from a #Kubernetes deployment - all with a simple #Python application built with FastAPI and SQLAlchemy. buff.ly/3xFq8Kb

🚀 Today on Velocity Byte Sized: Learn about high availability in #Kubernetes with ReplicaSets and Deployments in this "byte sized" chunk. Watch the 10-minute walkthrough: buff.ly/49qIiwm #developers #IDE

🚀 Today on Velocity #ByteSized: Let’s discuss how to write and debug code in your local #IDE, while it automatically syncs to your cluster, in order to develop directly in your cluster while working from your local machine: buff.ly/3ITz0xX #kubernetes #developers

Had a great chat with Paul at #Kubecon #EU 2024 #Paris about #CloudDevelopment today! 🚀☁️💻 Key insight: the power of remote environments to rapidly develop, test, and debug code. No more local machine limitations! Cloud development enables efficient collaboration, easy scalability, and the flexibility to "poke around" as we build more resilient apps. 💪 What cloud dev trends have inspired you lately? #velocity #kubecon #rapiddevelopment #clouddevelopment #RemoteDev #developers @velocitytechhq

I'm pumped for #KubeCon EU 2024 in #Paris - it'll be my 3rd #KubeCon! If you're attending, swing by booth #B32. I'd love to chat about cloud development and building/deploying applications on #Kubernetes. It's a great opportunity to meet other like-minded people in the Kubernetes community. 🇫🇷☁️ #velocity #kubecon #kubernetes #eu #paris #developers #cloud #k8s #devex #CNCF

Check out the Velocity Byte Sized series of small guides and posts about development - this one is focused on mixing #Python, #Kafka, #Neon, #Kubernetes and #Velocity

We remain committed to our partnership with OpenAI and have confidence in our product roadmap, our ability to continue to innovate with everything we announced at Microsoft Ignite, and in continuing to support our customers and partners. We look forward to getting to know Emmett Shear and OAI's new leadership team and working with them. And we’re extremely excited to share the news that Sam Altman and Greg Brockman, together with colleagues, will be joining Microsoft to lead a new advanced AI research team. We look forward to moving quickly to provide them with the resources needed for their success.

You probably heard the news regarding Shani Louk. I do - however - want to explain to you what happened. This is Shani, by the way, 23 years old. A German-Israeli. She went to Nova music festival for a party. She’s the one who was filmed on Saturday laying face down on a pickup truck. Naked. Her leg terribly broken and dislocated. The crowd was seen spitting on her, poking her body. She was proudly paraded through Gaza. In recent days, the IDF conducted missions in the north part of Gaza and the area surrounding the border. Parts of Shani’s skull were found. She was partly beheaded. After lab and DNA tests a panel of experts reached a conclusion that she was murdered and a formal message was delivered to her family. She could have been your sister, your daughter, she could have been you. Shani’s body is still being held by Hamas in Gaza along with 228 other hostages. The Red Cross is still not allowed to see them, no information regarding their condition was shared. #BringThemHomeNow

I see there's some confusion labeling Hamas terrorists as Palestinian 'Freedom Fighters'. So let me help you, Freedom Fighters don't: - Kidnap children - Rape women - Murder elderly - Burn entire families This isn't a heroic resistance; it's sheer terrorism. #hamas_is_ISIS

One thing that I started doing at OpenAI is that I created a policy for myself to be *100% transparent* with my manager about everything. It seems obvious and weird to say aloud, but I bet most people don’t actually do this. But once I started doing it, I realized there are a lot of benefits. For example, when managers ask people “how is your week”, many people may say “good” and then dive into a report of what they did that week. Some people may be shy about any struggles they may be having. I’ll be fully transparent about everything—X is good, I’m blocked on Y because of Z, I like A and B but I wish C could be better. I tend to use my 1-1s to talk about bigger picture stuff instead of technical details since that’s where managers can help the most (I’m experienced enough to not require technical guidance most of the time). Another example is when managers ask “is there anything the team could do better for you.” It feels odd to critique the person in charge of your career, but I am very blunt about this, while being clear about the broader context (here are 5 things that I don’t like, but I’m so grateful to be here working with you). I also communicate that I have no expectation of them solving my problems—I’m simply giving them information so they can appropriately triage it into their priority list. My guess is that managers actually appreciate this; it’s much worse to learn that someone doesn’t like their role via a conversation after they announce that they’re leaving. For some people it’s also awkward to ask for things from your manager (e.g., can I spend some time on this, promotions, becoming a manager, etc). When I’m serious about something I bring it up very early with some hedging (e.g., I know it can be far off but I want to work towards X). This helps your manager give feedback early and help keep you on track. The biggest benefit for me, though, is that verbalizing this to another person forces honesty on myself—have I been working hard enough to deserve what I’m asking for? Am I performing at my best recently? If not, why not? Finally, being transparent with your manager can also open the door for them to be more transparent about your performance. I try to open this conversation by regularly asking “what can I do better”. In the past I’ve felt that some managers are too nice to deliver negative feedback. But if you’ve opened up to your manager about everything, they might be more willing to give feedback. For example, I’m now more aware that I have less SWE experience than most people my level, and I should work extra hard on that to catch up. Of course, all this going well is conditional on working in a healthy company and having a decent manager. I’ve been super fortunate with the managers I’ve had at OpenAI. But if that’s not the case, some honesty may be even more revealing—do you want to keep working for someone who doesn’t ask for feedback, or who doesn’t take your problems seriously? A quote I heard from someone else is “every meeting between you and your manager should be slightly awkward.” I think it captures this idea pretty well—you should talk about personal and important things, and transparency is the first step.

Thinking about multi-tenancy in K8s? Check out this recent @Velocity webinar on the topic! buff.ly/3VFPtdP #kubernetes #devops

A really nice post about getting the most out of your #HelmCharts :) buff.ly/3VXlXAN

Pythonistas turned Gophers will like this: an IPython-like tool for @golang! buff.ly/2m97ouq #pythonprogramming #gopher #golang

Rust’s recent addition to the @Linux_Kernel is super exciting! I’m looking forward to seeing @rustlang evolve as a result. buff.ly/3UXI1Lg #rustlang

Curious about what 2023 has in store for software development?✨From #microservices to #devex to #lowcode #nocode, here are 🔟 trends that we predict will be at the forefront in the coming year. hubs.li/Q01vFrnp0

Event-driven architectures aren’t just for style points. Learn about the practical advantages of this micro-service networking approach with a simple, #Redis-based example. Check it out on the #Velocity blog ➡️ hubs.li/Q01sm3Jc0

In part two of our Passing Secrets with #Kubernetes series, we're using @HashiCorp #Vault to store secrets. Check it out in the #Velocity blog! hubs.li/Q01rr39j0