"How many partitions should we use for this topic?"
Get it wrong, and you pay for it for years. Databases abstracted this away decades agoβnow streaming is finally having its database moment.
Landing in StreamNative Pulsar 5.0, we are making topics work like tables:
π Scalable Topics: No more guessing partition counts. The system auto-splits/merges range segments based on load while keeping key ordering perfectly intact.
π§ ORM-Style APIs: 3 purpose-named interfaces (StreamConsumer, QueueConsumer, CheckpointConsumer) turn invalid runtime operations into compile-time errors.
π Silent Rebalances: Consumer disconnects hit a grace period first. Rebalances become background infrastructure events, not application fire drills.
Streaming infrastructure shouldn't demand so much of its users. Catch up on the full paradigm shift.
π Link to the blog in the replies!