A Large-Scale Study on the Development and Issues of Multi-Agent AI Systems
Multi-agent AI frameworks are everywhere. But how are they actually built and maintained?
The default assumption is that popular frameworks like LangChain, AutoGen, and CrewAI are mature, well-maintained systems.
But nobody had actually measured this at scale.
This new research presents the first large-scale empirical study of open-source multi-agent systems, analyzing over 42,000 unique commits and 4,700 resolved issues across eight leading frameworks: AutoGen, CrewAI, Haystack, LangChain, Letta, LlamaIndex, Semantic Kernel, and SuperAGI.
What did they find?
Three distinct development profiles emerge. Sustained systems like Haystack show consistent growth with periodic refactoring.
Steady systems like Semantic Kernel maintain balanced activity. Burst-driven systems like SuperAGI undergo intense but short-lived development spikes followed by minimal engagement.
The commit analysis reveals surprising priorities:
- Perfective commits (feature enhancement) constitute 40.8% of all changes.
- Corrective maintenance (bug fixes) accounts for only 27.4%.
- Adaptive updates sit at 24.3%.
- These systems prioritize building new features over fixing existing problems.
The issue landscape tells a different story:
- Bugs dominate at 22% of all issues.
- Infrastructure problems account for 14%.
- Agent coordination challenges represent 10%.
- Median resolution times range from under one day (LlamaIndex) to over two weeks (Semantic Kernel), with high variance pointing to uneven maintenance capacity.
Issue reporting increased sharply across all frameworks starting in 2023, marking a broader inflection point aligned with rising interest in AI agents. LangChain alone has over 14,000 commits.
The MAS ecosystem is characterized by fast iteration and feature-oriented development, but uneven maturity and maintenance practices. The momentum is real, but so is the fragility.
Paper:
arxiv.org/abs/2601.07136
Learn to build effective AI agents in our academy:
dair-ai.thinkific.com/