If you're just starting to track ML experiments, a well-maintained, easily discoverable spreadsheet suffices for a small team that runs a few experiments monthly. Like most mechanisms, in a few months, it will look nothing like how it started (and you'll probably move to Weights & Biases or AimStack). At the very least, track the following details: • ID: a moniker or a version number to identify the experiment • Headline: a single-line description of the experiment • Contacts: directly responsible individual(s) to contact if needed • Start and End Dates: to track the period during which the experiment is active • Hardware: specific hardware configuration used to run the experiment • Report URL: where the report and associated writeups can be found • Code URL: where code lives (e.g., a permalink to a notebook in Git) • Data URL: where relevant datasets can be found (the exact version) • Logs URL: where logs (e.g., training logs) can be found • Model URL: where model deployment artifacts can be found (again, the exact version) Assuming your experiments target a somewhat standard set of experimental factors (e.g., hyperparameters) and performance metrics, you may include them in the spreadsheet for comparison. Limit the spreadsheet to commonly used details, leaving sparsely used ones to the detailed experiment reports. Each report should also include — at the very least — the hypothesis to test, the null hypothesis, and performance notches (when possible) indicating baseline, oracle, and human-level performances. It also needs to describe how to reproduce the experiment. Reproduction steps need to be unambiguous to the extent that a new hire can run them and reproduce the results (within acceptable variances). Reproducibility, especially on GPUs, is extremely hard, but that's a post for another time. #ShippingMachineLearningSystemsBook

完整公司清单： 隐私安全平台： Privacy, safety and compliance platforms Protect enterprises from legal, compliance and privacy risks of integrating LLMs Commonbase Arklow Skyflow ProtectAI Pancake Data Credal AI Arthur Robust Intelligence Tillon AI Monitaur Lega AI ConductorAI Radiant AI Guardrails Rebuff Cranium BerriAI Cadea AIShield Adversa Lakera Fairly Calypso AI InsightCircle.ai Nomic Sarus 数据接入和存储公司： Data Integration and Storage Platforms Connect enterprise data sources to LLMs Metal Chima Oxen LlamaIndex Langchain Scale Nucleus Tecton Psychic Unstructured Arcus Flower Baseplate Vectara AimStack Unstructured MindsDB Marqo Faiss Graphlit 中间件和应用层建设工具 Middleware and application builders Build production-grade LLM-powered internal tools, features and apps Anarchy GradientJ Chidori AutoInfra Vellum Scale Enterprise Generative AI HumanLoop Retool (specifically, their OpenAI integration) Tennr Stack AI Fixie LastMileAI Layer Respell.ai Stochastic pyq Kinesys Dust Fine-Tuner 数据监控和评估 Observability, Monitoring and Evaluation Monitor, trouble-shoot and rollback LLM-powered apps and features PromptLayer Parea Giskard Arize Gantry e2b Weights & Biases LMQL Helicone Fiddler Aporia HoneyHive Portkey Superwise Autoblocks Prediction Guard MLFlow WhyLabs Patronus.ai TruLens Databricks UpTrain AI Babylon AI Langfuse Lightup MonteCarlo 开发者平台 Developer Platforms Vector databases Represent data with embeddings to perform ML tasks and train models on that data and store it properly Pinecone Weaviate LanceDB Ziliz ActiveLoop Chroma Qdrant Milvus pgvector 模型开发和部署 Model development, hosting and deployment Develop, deploy and host ML models cheaply, efficiently, and securely Brev Texel.ai Unify Trainy Modal BaseTen Anarchy Replicate Poplar Chart Hugging Face Foundry PyTorch Cerebrium Together Ray Stochastic Banana GooseAI Lambda Labs OctoML airOps LGN Graft Skymel Seldon PrefectHQ

This is an awesome thread on how to use @aimstackio with LlamaIndex. 🎯🦙 Building an data-driven LLM app is complex. We abstract away that complexity, but our users have asked for greater transparency/visibility into what's going on under the hood. AimStack helps solve that.

Aimstack template for Huggingface Allows you to track all AI metadata changes and experiments in your space

Hope this helps, please let us know @aimstackio if you bumpt into other issues / blockers. Would be happy to help you out all the way though 🙌🚀

@aimstack how do I make AimCallback() log to an existing run instead of having it create its own?

Accelerate your Prophet model training and evaluation with Aim's intuitive UI and powerful search functionality. Check out this newly published article to learn how to integrate Aim with Prophet experiments effortlessly: bit.ly/3mTT0sR #aimstack #prophet #opensource

EVN Disrupt podcast: Gevorg Soghomonyan, co-founder/CEO of AimStack, an open-source machine learning experiment tracker speaks to @satourian about how AimStack tackles the problem of tracking & visualizing ML metadata, the current MLOps landscape more. evnreport.com/podcasts/evn-d…

The Aim UI Kit v2 is coming soon! Already created 15 accessible components with improved performance and much superior experience. Importantly, we are going to ship the dark mode Aim 🚀 @aimstackio #aimstack #MLOps #uiux

よさげ Exploring MLflow experiments with a powerful UI | by Gor Arakelyan | AimStack | Jan, 2023 | Medium medium.com/aimstack/explorin…

Great news!! We are finalizing pre-binned histograms support💥 Provide distribution values, Aim will visualize and display it 📊 Who is waiting? Countdown begins... @aimstackio #aimstack

I wonder how many lines can be rendered in line chart? . . . . . Rendered over 16k lines with 100 x-values with Aim. @aimstackio #aimstack #d3js

Hi there 🙌 Aim 3.14.0 is packed with new features. Now it's possible to navigate through your last three queries from the Dashboard page. Do you want more time travel? @aimstackio #aimstack #MLOps #OpenSource

Good news 🤩 @aimstackio Now you can pin the chart tooltip on the top or the bottom of the panel to not impede looking at the visualization starting from Aim v3.14.0. Just click on the top buttons of the chart popover. #aimstack #MLOps

Data Scientists Everywhere: Hey, MLFlow’s UI is pretty good AimStack: Let’s Supercharge it and make it even better aimstack.io/aimlflow

📣 Aim 3.0.0 is now live 📣 After so many iterations, Aim has a new look and a new engine! 🌟 We have also published the Aim roadmap for the next 3 months 😊 blog: bit.ly/3vzh2cA AimStack: aimstack.io/ #machinelearning #pytorch #tensorboard #nlp