Which data validation library do you use? - pydantic - msgspec - attrs - other

We built a 𝐭𝐲𝐩𝐞-𝐠𝐮𝐢𝐝𝐞𝐝 𝐬𝐞𝐫𝐝𝐞 𝐬𝐲𝐬𝐭𝐞𝐦. It reads your Python type hints and picks the right serializer automatically. 🚀 Here's how we did it: CocoIndex has a Rust core and a Python SDK. Under the hood, the engine serializes and deserializes Python objects constantly — caching function results, persisting pipeline state, tracking changes — with data crossing the Rust/Python boundary. • 𝐦𝐬𝐠𝐬𝐩𝐞𝐜 for dataclasses, NamedTuples, primitives — 3–15x smaller than pickle • 𝐏𝐲𝐝𝐚𝐧𝐭𝐢𝐜 serde for BaseModel subclasses — no pickle, full validation preserved • 𝐏𝐢𝐜𝐤𝐥𝐞 only when you explicitly opt in — still through a restricted unpickler A single routing byte at the start of each payload dispatches to the right engine. Your return type annotation 𝒊𝒔 the deserialization schema. Full deep dive (with union type edge cases worth knowing about) 👇 cocoindex.io/blogs/type-guid… #CocoIndex #Python #OpenSource #DataEngineering #AI #DataInfrastructure #Developer

See the Dhi repository: "For basic struct parsing without validation, msgspec performs at ~10M/sec, while dhi with validation performs at ~5.7M/sec" For extra parameters I built github.com/msgflux/msgspec-e…

it uses dhi which is even faster than msgspec!(and has more fields that it supports too)

msgspec go brr

just realized i could just use msgspec structs for everything in my game. this will allow to remove a lot of code for wire-only structs. currently we use dataclasses for runtime and msgspec for serialization (replays, netplay, snapshots, traces). and what's funny is this actually works faster and gives us some very good features like tagged union types. check out this library, it's great! jcristharif.com/msgspec/

You use msgspec.Struct like you'd use Pydantic for your typing. You use msgspec JSON serialization that is faster than ujson and handles types like datetime, UUID, bytes and Decimal automatically. Take requests as Struct already validated and converted, return as responses, WS.

とても素敵なツール // koxudaxi/datamodel-code-generator: Python data model generator (Pydantic, dataclasses, TypedDict, msgspec) from OpenAPI, JSON Schema, GraphQL, and raw data (JSON/YAML/CSV). github.com/koxudaxi/datamode… #pyconshizu

[Projects] FarhanAliRaza/django-rapid Msgspec based serialization for Django. #djangonews github.com/FarhanAliRaza/dja…

takopi v0.21.0 - /ctx now works in any chat mode, not just topics - takopi config command for easy configuration - telegram parsing rewritten to msgspec - test coverage improved to 80%, added mutmut testing takopi.dev/

Vibe Coding 哲学方法论提效工具箱（Python版） 0. 总体作业流（建议默认流程） 1) 现象卡片（现象/意图/情境/边界） -> 清零脑补 2) 规格化（类型 schema 错误语义 不变式） -> 可机器检查 3) 检查器（单测 性质测试 lint 类型检查 关键断言） -> 可证伪 4) 最小实现（main path） -> 快速跑通 5) 反例驱动（Hypothesis/边界/差分/基准） -> 找到失败模式 6) 收敛重构（删复杂度、固化概念、稳定接口、补文档） -> 可维护 推荐底座（Python） - ruff black pyright(或 mypy) pytest hypothesis pydantic(msgspec可替代)

takopi schemas are also a good source for how to decode jsonl events of popular agents. i put a lot of effort into making the definitions as accurate as possible. they are written using msgspec struct types. it’s the best library for this, much faster than loading json and transforming it. you go straight from bytes to a decoded python objects.

takopi v0.7.0 - msgspec schemas for jsonl stream decoding - much prettier logging with structlog github.com/banteg/takopi

msgspec also lets you defer some parsing by leaving it as bytes to use a custom parser if you want. you can GPT a function to do list[list[str]] -> ndarray[float] muuuch faster than the parser can.

msgspec gives you a lot more flexibility, closer to what you'd get w/ pydantic but with less overhead. If you already have a dict though less argument to bother?

Should disclaimer the screenshot is 1 years old (curious to repeat on 3.14 though now if I find some time though) and not 100% optimized. Its with structs yeah, msgspec is significantly worse than orjson w/o the structs as I recall (and as you'd expect).

is this with structs for msgspec? I ask knowing I definitely won’t refactor our code to use orjson for a few nanos

not recently! from memory msgspec only outperforms if you use structs

Have you benchmarked msgspec against orjson?