Magick is finally in a state where we can build things like BabyAGI within an hour. You can watch the graph executing in real time, watch the thought process as it executes, and understand the flow. This is unbelievably cool.

We have been quiet for a while. We have been heads down building something new. We are building an Agent framework, with our Magick IDE fully open sourced at the core of it. We invite you to join in and let us know what would be most useful! github.com/Oneirocom/Magick/…

Amazing work coming down the pipeline. This makes it all worth it.

We have been excited about this for a long time, and we finally have them back in V2. Sub-spells. Vastly improved and better than ever. In this case, we brought in the amazing design pattern created by @unity with the subgraphs in visual scripting.

We have sub-spells. Nested graphs are coming to Magick. The next step will be sub-spells as tools. The socket inputs and outputs on the sub-spell are given descriptions, and we can transform them into tool schemas. Then we parse and call the sub-spell, returning the result. This is going to open up all kinds of possibilities.

New feature incoming. Variable events. Handle side effects and other processes when a variable changes. And until we get custom events in, it does basically the same thing.

We just pushed out a major update to Magick. We integrated @keywordsai as our LLM proxy, providing our users with over a hundred models to use. We have improved stability for large graphs, with UI performance working at well over 300 nodes. We will be announcing more very soon.

OpenAI is expected to demo a real-time voice assistant tomorrow. What does it take to deliver an immersive, or even magical experience? Almost all voice AI go through 3 stages: 1. Speech recognition or "ASR": audio -> text1, think Whisper; 2. LLM that plans what to say next: text1 -> text2; 3. Speech synthesis or "TTS": text2 -> audio, think ElevenLabs or VALL-E. Last year, I made the figure below to show how to make Siri/Alexa 10x better. However, naively going through 3 stages results in huge latency. User experience falls off the cliff if we have to wait 5 seconds for *each* reply. It breaks the immersion and feels lifeless even if the synthesized audio itself sounds real. Natural dialogues fundamentally don't work like this. We humans > think about what to say next at the same time as we listen & speak; > inject "yes, hmm, huh" at appropriate moments; > predict when the other person finishes and immediately take over; > decide to talk over the other person organically, without being offensive; > handle interruptions gracefully. Currently, AI assistants either cannot be interrupted (super frustrating) or simply stop when they detect an audio event and lose train of thought; > engage in group chat. We are so good at multi-agent conversations. It's not as simple as making each of the 3 neural nets faster, sequentially. Solving real-time dialogue requires us to rethink the whole stack, overlap each component as much as possible, and learn how to make interventions in real time. Or perhaps even better - just have 1 NN mapping audio to audio. End-to-end always wins. I'll sketch out how to design such a model and its training pipeline. Meanwhile, let's wait and see how far OpenAI pushes it!

Check out our brand-new course on Building Agentic RAG with LlamaIndex with @DeepLearningAI @AndrewYNg 🔥 Learn how to build an autonomous research assistant that can answer complex questions over multiple documents - handle a much broader range of inputs than a standard RAG pipeline! 📖🕵️ The course helps you learn how to build an agent ingredient by ingredient from a standard RAG pipeline. Once you're done with the course check out the following @llama_index resources to take your agents to the next level 👇 Building a custom agent: docs.llamaindex.ai/en/stable… Multi-document agents (diagram by @clusteredbytes): docs.llamaindex.ai/en/stable… Course: deeplearning.ai/short-course…

Another small UX improvement. Hiding unused sockets. It isn't a full node collapse, which I want to do next, but that will have to wait. In demo crunch time.

I love adding these little UX improvements that make my experience building way better. Renaming a nodes title goes a long way to legibility. Even more so possibly than comments.

Getting closer...

I have just decided that I love multi-gates. I find this incredibly satisfying. Using this in the demo I am working on, and will have to document this.

Even more satisfying.

let me cook @MagickML

We have docs! :D docs.magickml.com

New things coming down the pipeline.

Can’t wait to get this into Magick.

Been talking lots about knowledge graphs and LLMs, specifically for autonomous agents, recently. Organized some thought here if you want to dig in 👇 KG 🤝 LLM 🧵 (a thread)

The most epic AI panel in a while! We at NVIDIA have gathered ALL 8 authors of "Attention is All You Need" for a panel at GTC, hosted by none other than the GOAT himself, Jensen Huang. In 2017, 8 researchers had a flash of genius and invented Transformer, the seminal work that transformed AI once and for all. How did they come up with the idea? What challenges did they face? What surprised them in the following years? What have they been up to these days? How do they envision a post-transformer future? Thanks @maggie_albrecht for making this happen! I'm so glad to play a part in reaching out to some of the coauthors. Join us for a behind-the-scenes look at modern AI's greatest moment. Mar. 20, 11 am PDT, at San Jose McEnery Convention Center, NVIDIA GTC: nvidia.com/gtc/session-catal…

Brad Templeton asks if it is a copyright violation to recite a poem or song because you have a copy of it in your brain