Introducing your newborn to the world is a feeling every parent knows. That is exactly how I feel today as we unveil @autohealai, the AI for production engineering. Our mission is to empower engineering teams to build unbreakable customer experiences. @utkarshohm, @puneet_sar, and I believe AI’s true value lies in turning engineers into superheroes especially in production. Production has always been that one SDLC phase where engineering teams feel they have the least control. We are changing that permanently, starting with the production-facing roles of SRE and Support Engineering. To do this, we are solving the "production trifecta" for the enterprise: - Production Context Graph - Security & Governance - Unified Incident Management We are building at the frontier of agentic AI while partnering deeply with customers on their AI transformation journeys. Read more here: autoheal.ai/blog/introducing…

Shoutout to @airindia staff Sumit Tarun and sahir for going out of their way and hustling with multiple agencies to retrieve my phone when I forgot it at security check. I couldn’t deboard so they brought it to me on the flight. Airindia staff helped me when most airlines wouldnt

Been excited about this since Shreya told me about it! It’s the impending shift left of evals!

Am so hesitant to add readme file in code dirs fearing coding agent will update code but not doc at time t and stale docs will throw off the agent at time t 1. precom check that keeps readme updated? any solutions @dosu_ai @cognition_labs @augmentcode @coderabbitai @AnthropicAI?

The easiest way to get started with #claudecode or @cursor_ai agent on an existing code base is to ask it to configure/ improve pre-commit check, run it and fix all the failures. Did it for all my repos, programming with only agents now

🎧 New SSB Pod w/ @utkarshohm - Director of AI @thoughtspot 🎧 - Applying LLMs to High Fidelity Analytics Use Cases - Challenges with using Text to SQL model output - Data Security for LLM Enabled Products - Customer Outcomes from Sage Stream on X or other platforms (🧵)

"How is LLaMa.cpp possible?" great post by @finbarrtimbers finbarr.ca/how-is-llama-cpp-… llama.cpp surprised many people (myself included) with how quickly you can run large LLMs on small computers, e.g. 7B runs @ ~16 tok/s on a MacBook. Wait don't you need supercomputers to work with LLMs? TLDR at batch_size=1 (i.e. just generating a single stream of prediction on your computer), the inference is super duper memory-bound. The on-chip compute units are twiddling their thumbs while sucking model weights through a straw from DRAM. Every individual weight that is expensively loaded from DRAM onto the chip is only used for a single instant multiply to process each new input token. So the stat to look at is not FLOPS but the memory bandwidth. Let's take a look: A100: 1935 GB/s memory bandwidth, 1248 TOPS MacBook M2: 100 GB/s, 7 TFLOPS The compute is ~200X but the memory bandwidth only ~20X. So the little M2 chip that could will only be about ~20X slower than a mighty A100. This is ~10X faster than you might naively expect just looking at ops. The situation becomes a lot more different when you inference at a very high batch size (e.g. ~160 ), such as when you're hosting an LLM engine simultaneously serving a lot of parallel requests. Or in training, where you aren't forced to go serially token by token and can parallelize across both batch and time dimension, because the next token targets (labels) are known. In these cases, once you load the weights into on-chip cache and pay that large fixed cost, you can re-use them across many input examples and reach ~50% utilization, actually making those FLOPS count. So TLDR why is LLM inference surprisingly fast on your MacBook? If all you want to do is batch 1 inference (i.e. a single "stream" of generation), only the memory bandwidth matters. And the memory bandwidth gap between chips is a lot smaller, and has been a lot harder to scale compared to flops. supplemental figure medium.com/riselab/ai-and-me…

If you clone on correct answers that aren't in the knowledge graph, you're teaching the net to hallucinate. If you say "don't know" to answers that are in fact in the knowledge graph, then you're teaching the net to withhold information unnecessarily. 2/

Finally a website for LLM security! Which countries and businesses have banned #ChatGPT Reminds me of the covid dashboard days.

Is Falcon really better than LLaMA? Short take: probably not. Longer take: we reproduced LLaMA 65B eval on MMLU and we got 61.4, close to the official number (63.4), much higher than its Open LLM Leaderboard number (48.8), and clearly higher than Falcon (52.7). Code and prompt open-sourced at github.com/FranxYao/chain-of… No fancy prompting engineering, no fancy decoding, everything by default. ---- Full story: On OpenLLM Leaderboard (huggingface.co/spaces/Huggin…), Falcon is the top 1, suppressing LLaMA, and promoted by @Thom_Wolf (twitter.com/Thom_Wolf/status…) Yet later @karpathy expressed concern about why on Open LLM Leaderboard, the LLaMA 65B score is significantly lower than official (48.8 v.s. 63.4), see twitter.com/karpathy/status/… We figure that a simple quick open-sourced evaluation script on LLaMA 65B would clarify, so we just did it github.com/FranxYao/chain-of… Again, everything is default, official MMLU prompt, no fancy prompt engineering, no fancy decoding. LLaMA 65B simply can do it. We encourage everyone to try the eval script out. This result makes us continue to hold the belief that the best bet of open-source community to get close to GPT-3.5 is to do RLHF on LLaMA 65B, per our previous discovery in Chain-of-thought Hub arxiv.org/abs/2305.17306 Yet we do not intend to raise wars between LLaMA and Falcon -- both are great open-sourced models and have made significant contribution to the field! Falcon also have the advantage of a easier license, which also gives its great potential to be awesome! 🍻🍻

* Hiring five engineers for one year costs a million dollars. * Your observability bill(s) should come to about 20-25% of your infrastructure bill (combined) * If the tool costs >$30k/year, you can prob get a discount what other budget rules of thumb do you have, as a manager?

Apple's Object Capture now runs *on-device*! 💥 😳 This is a game changer as as any app can now integrate 3D scanning *for free*, and you can see where this is going at next WWDCs... developer.apple.com/videos/p…

It's fascinating that crowd sourced openly available knowledge created in last 20 years in wikipedia or stack exchange or arxiv or github is as large as that in all books ever written (and preserved)! Source: redpajama dataset