.@PareaAI also looks like a good LLM monitoring tool and is open source

How do you detect unreliable behavior of your LLM app? Recently, we talked to the team at @sixfoldai and they shared with us a simple, yet powerful way to assess the reliability of their LLM app using @PareaAI. More about how they test their risk assessment AI solution for insurance underwriters in the article in the thread

Saturdays are for doc upgrades

🚀 New deep dive notebook on @PareaAI experiments and LLM evals 📝🔬. I cover some of the key functionalities illustrating the power and flexibility of our API. 🔽 Link in comments 🔽

@cohere 's actually pretty awesome. More folks should be exploring their models. @PareaAI , now has auto-instrumentation for the Cohere py sdk 🚀

Also, learn more about the research behind each here: docs.parea.ai/blog/eval-metr…

There are so many “black box” evals that force users to instantiate eval classes. Never fully understood this. At @PareaAI we see evals as just functions. You can copy the source code and modify as you see fit, all OSS and based on latest research. Check these out👇🏾

📝 Updated integration docs ⭐️ Checkout @PareaAI's updated docs to automatically trace apps powered by @LangChain, instructor by @jxnlco, @LiteLLM, DSPy by @lateinteraction, SGLang by @lmsysorg, and @triggerdotdev. Docs: docs.parea.ai/integrations/o…

Day 1 support for llama 3.1 via @FireworksAI_HQ in @PareaAI's playground! 🧨🦙

And to help you understand what's going on, we integrate with observability platforms like @ArizePhoenix, @langchain's LangSmith, @langfuse, @PareaAI, and @lunary_hq so you can explore the experiments that zenbase/core automates. Cookbooks here: github.com/zenbase-ai/core/t…

Def agree this could be great. Probably best if you can train the router yourself. @anyscalecompute's RouterLLM tracing support with @PareaAI

With the latest @GroqInc models for tool calling, we figured it was time to make Groq available across @PareaAI's playground and SDK's. Be on the lookout for an updated tool-calling benchmark, OpenAI v Claude v Groq!

📝 Updated self-deployment docs ⭐️ Deploy @PareaAI on-prem via @Docker in 4 steps: 1. Clone the repo 2. Specify organization slug 3. Pull docker images & run them 4. Point SDK backend URL to self-deployed backend URL 🔗 -> 🧵

There have been so many new models lately. Most recently, @MistralAI 's codestral-mamba. I figured it'd be great to highlight how to use @PareaAI for Regression Testing. Check out the Notebook below, where I test codestral-latest vs mamba on LeetCode questions. 👇

At this point I could probably have an llm monitor the top foundation model providers and then produce a PR for me that adds any new models to @PareaAI the moment they launch.

Blog: python.useinstructor.com/blo…

If you use structured outputs with Instructor, track validation errors instantly with @PareaAI. Concretely, the integration automatically: - groups any LLM call due to retries together under a single trace - tracks any field which failed validation with the respective error message - visualizes validation error count over time Instrument calls made via the Instructor client by adding two lines: p = Parea(api_key="PAREA_API_KEY") p.wrap_openai_client(client, "instructor") Read the full blog post on the instructor docs in the 🧵

Moving from demos to production-ready LLM apps can be challenging. In this post, I outline a practical workflow to help teams make this transition, focusing on: - Hypothesis testing - Dataset creation - Effective evals - Experimentation Full post here: zurl.co/27Ad

This method is powered by DSPy from @lateinteraction and inspired by the work of @sh_reya: arxiv.org/pdf/2404.12272 arxiv.org/pdf/2401.03038 Also, thanks to @eugeneyan sharing JudgeBench: arxiv.org/abs/2406.18403

Blog: docs.parea.ai/blog/self-impr… Docs: docs.parea.ai/manual-review/…