🤔 How do we know if an LLM is good enough? In this session of the Agentic AI Bootcamp, instructor Raja Iqbal explored the nuances of LLM evaluation — a critical step for building trust, reliability, and ethical alignment in AI systems. We examined why evaluation matters: LLMs are probabilistic, producing variable outputs depending on prompts, temperature, and context. Evaluation ensures responses remain accurate, safe, and consistent across use cases. The session covered the challenges of measuring quality: subjectivity in tone, helpfulness, and bias makes deterministic scoring impossible. Standard benchmarks like MMLU, BIG-Bench Hard, and HotpotQA provide reference points, while traditional metrics such as BLEU, ROUGE, and BERTScore offer complementary dimensions of assessment. Finally, we explored RAGAs, a framework designed for Retrieval-Augmented Generation (RAG) systems. RAGAs evaluates both retrieval and generation, measuring faithfulness, relevance, and precision/recall — enabling fine-grained, production-ready evaluation of complex AI workflows. 📅 Want to join live? Register now for the upcoming Agentic AI Bootcamp happening on Nov 25th. Don’t miss your chance to build, test, and evaluate intelligent agents! hubs.la/Q03SRlHx0 Evaluation is more than a score — it’s the foundation for trustworthy and reliable AI. #LLMEvaluation #AITrust #RAG #RAGAs #AgenticAI #LanguageModels #BenchmarkingAI #MMLU #BIGBenchHard #HotpotQA #BLEU #ROUGE #BERTScore #AIWorkflows #ResponsibleAI

What's the smallest decision you ever made that accidentally changed the entire course of your life?

This is empirically incorrect. Hundreds of thousands of fast.ai students have learned the required math for ML as they go. By *far* the biggest problem we've seen is from people who try to learn the math first. They learn the wrong stuff & have not context.

Our Python AI series is over, but you can still watch the videos, download the slides, and try out the code. Get it all from aka.ms/pythonai/resources

Harvard Professor reveals the 5-phase path every ML systems engineer follows but almost no one talks about. Completely free, continuously updated and collaboratively developed on GitHub. Link in comments

Things I learned this year as an ML Engineer: (DON'T MISS) - Focus on data; the solution lies within it. - XGBoost outperforms many classic ML algorithms and excels at time-series. - UV is the best tool for Python package management. - For applied ML, build first, then read research papers. - Math and statistics/probability are essential skills. - Caching is critical for ML projects. - Agentic AI frameworks aren’t needed for LLM function calling. - FastAPI and PyTorch are a powerful duo. - When using ChatGPT, provide input and problem statements. Brainstorm pipelines, don’t ask for code. - Instruct ChatGPT: “You are a 10 year ML Engineer expert in XYZ domain,” then share the problem. - Work with quantized LLMs. - Reinforcement Learning will outlast LLMs in relevance. - Deploy models first, then improve iteratively. - Speed currently outweighs accuracy; I can handle errors but not slow inference. - Data Engineering > AI/ML Engineering. - Use AI to learn Next.js/React.js for high returns. More insights to come. - Apple M-Series chips are powerful but doesn't support CUDA libraries at all. - MLOps is a must skill for ML Engineers and demand is very high. What's your experience in ML this year?

System Design, particularly in ML can have a varying number of requirements- but the ideas generally fall into a number of categories. Depending upon context, specifications change, but baseline components remain unchanged. Watch this video to understand how these interviews are tackled. Great strategy and execution shared.

This ChatGPT cheatsheet saved me 10 hours per week:

Holy Shit... Google, Microsoft, OpenAI, and the biggest companies and agencies shared complete AI Agent playbooks. These are battle-tested formulas for building agents, the major bottlenecks in the industry, and the common patterns of what does and doesn't work in AI products. 1. Google - Startup Technical Guide for AI Agents 2. Microsoft: Agent Governance whitepaper 3. Cohere: Building Enterprise Agents 4. Amazon Web Services (AWS): An Executive's Guide to Agentic AI 5. Deloitte: Unlocking the right Agentic AI Use cases 6. McKinsey & Company: Seizing the Agentic AI Advantage 7. KPMG: The Agentic AI Advantage 8. Capgemini: The rise of Agentic AI 9. Accenture: Technology vision 2025 10. OpenAI - AI in Enterprise 11. Google: AI Agent Handbook 12. BCGX: AI Agents and MCP 13. Thomson Reuters and Reuters: Agentic AI 101 14. ServiceNow : Enterprise AI Maturity Index 15. Infosys - Tech Navigator Agentic Enterprise Playbook You don't need to read them all, just what is relevant to you, as a manager, AI developer, solo app creator, etc. If you want this list, subscribe to my newsletter, and you will receive it right away: hesamation.com/newsletter/

50 LLM Projects with Source Code to Become a Pro 1. Beginner-Level LLM Projects → Text Summarizer using OpenAI API → Chatbot for Customer Support → Sentiment Analysis with GPT Models → Resume Optimizer using LLMs → Product Description Generator → AI-Powered Grammar Corrector → Email Auto-Responder using GPT → YouTube Title & Description Generator → News Headline Summarizer → Blog Post Topic Generator 2. Intermediate LLM Projects → Legal Document Summarizer → AI-based Code Comment Generator → SEO Keyword Optimizer using LLM → AI-driven Study Notes Generator → Research Paper Simplifier → Tweet Sentiment & Engagement Predictor → Interview Question Generator → Personalized Learning Assistant → AI-based Ad Copy Generator → Legal Contract Review Assistant 3. Developer-Focused LLM Projects → Code Explanation Tool using LLM → Debugging Assistant using GPT and LangChain → AI Code Reviewer with OpenAI API → AI-based Test Case Generator → SQL Query Generator using LLM → Python Script Generator from Prompts → StackOverflow Question Answerer → Documentation Auto-Writer → AI Git Commit Message Generator → Programming Concept Tutor 4. Data and Research Applications → Data Insights Chatbot for CSV/Excel files → AI Research Assistant for Literature Review → Dataset Annotation Assistant → Paper Citation Generator → Data Cleaning Assistant using LLM → Scientific Abstract Generator → AI-Powered Survey Response Analyzer → Knowledge Base Chatbot using LangChain → Document Q&A System → Custom Retrieval-Augmented Generation (RAG) App 5. Advanced and Production-Ready LLM Projects → AI Content Management Dashboard → Multi-Agent Research Collaboration Tool → Personalized AI Writing Coach → AI-Powered Search Engine using Vector Databases → Voice-to-Text and Text-to-Insight Application → Financial Report Analyzer with LLM Integration → AI Chat Assistant for E-commerce Websites → Legal Research AI Assistant → LLM Fine-Tuning Pipeline for Custom Data → Full-Stack AI SaaS Platform using Next.js and OpenAI API Get the complete LLM Projects Handbook here: codewithdhanian.gumroad.com/…

great blog post going through some Deep-ML questions start to end!

If you're getting into ML, layer your learning, don’t try to get into everything at once. Start with Python, pandas/numpy, and just enough math (stats, probability, linear algebra) to get how the models work. Then move to ML basics mini projects to build intuition. Next, get really hands-on with real datasets. You can choose from any on my page. Focus on explaining your choices clearly. Then go deeper: PyTorch/TensorFlow → CNNs/RNNs → Transformers & RL. Balance theory with projects, that’s what gets you noticed.

if you're looking for the perfect way to break into LLMs - this is it! @karpathy released nanochat which is a great example of the entire LLM lifecycle: 1. training the tokenizer 2. pre-training on a large amount of diverse data 3. finetuning on conversational data to adapt for chat use cases 4. additional supervised finetuning (SFT) on higher-quality data safety 5. reinforcement learning to squeeze out a few more percentage points of performance additionally, you’ll learn how to log everything with @weights_biases - because logging and monitoring are essential for model training.

Early in my LLM journey, I spent hours experimenting with cookbooks, building RAG pipelines, designing agent workflows, and fine-tuning small models on free Colab GPUs. While today’s trend leans toward larger models and flashy demos to showcase capability, I still believe capable, accessible setups—using consumer-grade hardware and well-crafted recipes—can empower more developers to learn, build, and create real value at lower cost.