Make AI agents production-ready through independent evaluation and training.
Joined January 2024
- Tweets 99
- Following 14
- Followers 162
- Likes 159
37 Photos and videos
Pinned Tweet
Jan 26
Recently we built OTelBench – a benchmark to test how well LLMs handle OpenTelemetry instrumentation.
We tested 14 models. The best (Claude Opus 4.5) hit only 29%.
These weren't trick questions, just small subset of typical SRE tasks.
Link here:
quesma.com/blog/introducing-…
3
1,127
Quesma retweeted
Feb 27
AI Ghidra by NSA = reverse-engineering fun
I am speaking at @AITinkerers Warsaw, 4th Mar 2026.
One of my favorite event series - by and for the creators community.
Vibe-resurrecting an old game from binaries 👾 and vibe-hardware-ing a LED backpack 🎒🌈.
1
2
7
372
Quesma retweeted
Feb 10
Claude can code, but can it read machine code?
We gave AI agents access to Ghidra (a decompiler by the NSA) and tasked them with finding hidden backdoors in servers - working solely from binaries, without any access to source code.
See our BinaryAudit: quesma.com/blog/introducing-…
75
179
1,442
231,926
Quesma retweeted
Jan 25
Great to see the community releasing benchmarks in @harborframework now. These are invaluable resources for collectively building the most useful agents.
Jan 25
Last week @QuesmaOrg released “terminal-bench-sre-part-1” called OTelBench in Harbor.
Another releasing coming soon. Maybe even next week.
1
1
10
1,740
Quesma retweeted
Jan 9
I used to cite Gartner, now I quote @GergelyOrosz and his Pragmatic Engineer. Enjoy our new blog post:
quesma.com/blog/prompts-sour…
1
3
261
24 Nov 2025
Finally, an AI that can draw a map without getting lost. Nano Banana Pro uses tools to create factually correct infographics - and it's a game-changer.
quesma.com/blog/nano-banana-…
2
1
263
Quesma retweeted
5 Nov 2025
Postmortems are painful to write, especially this one. Sharing my startup Quesma journey so far.
quesma.com/blog/database-gat…
2
3
19
2,040
24 Oct 2025
Interesting use case for AWS Lambda that we explored: sandboxing AI-generated code.
We tried WebAssembly first but hit the wall. So, we scrapped our experiment for AWS Lambda with Docker containers in an isolated VPC.
Full writeup from @pmigdal:
awsfundamentals.com/blog/san…
24 Oct 2025
Lambda has tons of use cases, but one I've missed: using it as some kind of sandbox for running AI-generated code.
Lambda's isolation and scaling are a solid fit for this problem.
1
183
The security paradox of local LLMs - quesma.com/blog/local-llms-s… by @jakozaur at @QuesmaOrg
If you’re running a local LLM for privacy and security, you need to read this. Our research on gpt-oss-20b (for OpenAI’s Red‑Teaming Challenge) shows they are much more prone to being tricked than frontier models. When attackers prompt them to include vulnerabilities, local models comply with up to 95% success rate. These local models are smaller and less capable of recognizing when someone is trying to trick them.
#AISecurity #LLMSecurity #LocalLLM #GenAI #MLOps #ModelRisk #DataPrivacy #AIPrivacy #PromptInjection #AIThreats #AIGovernance #EdgeAI
4
8
350
18 Sep 2025
Can AI compile 22-year-old code? We built CompileBench to find out.
We know that LLMs can vibe-code or even win IOI, but what about dependency hell or legacy build systems?
(image based on XKCD 2347)
ALT Cartoon about dependency hell; tangled ‘dependencies’ making simple tasks complex.
1
4
194
18 Sep 2025
Cost-efficiency crown: @OpenAI.
Across difficulties, OpenAI models dominate the Pareto frontier of cost.
GPT-5-mini (high reasoning) is a great price/perf pick; GPT-4.1 is the fastest with solid wins.
ALT Scatter plot of success vs cost, highlighting OpenAI models.
1
2
137
18 Sep 2025
See the full ranking and every run (logs, commands, binaries), methodology & code:
▶️ compilebench.com
💻 github.com/QuesmaOrg/Compile…
📃 quesma.com/blog/introducing-…
3
101
9 May 2025
At #IcebergSummit 2025, Ryan Blue unveiled Iceberg beyond Java, plus the path to Table Spec V3 & forward to V4. Przemysław Delewski’s new blog covers Fokko Driesprong on Pylceberg, Matt Topol on Go, Julien Le Dem on modular DBs. Essential read for next-gen data platforms. Link👇
1
3
197
Quesma retweeted
24 Apr 2025
Everything is better when Kawaii 🌸🌸🌸:
Titanic survival rates with freshly-released Quesma Charts.
app.charts.quesma.com/s/20bv…
At @DataCouncilAI conference in Oakland with Jacek Migdał.
#dataViz @QuesmaOrg @jakozaur
2
4
311