Joined November 2018
1,879 Photos and videos
"How many partitions should we use for this topic?" Get it wrong, and you pay for it for years. Databases abstracted this away decades agoβ€”now streaming is finally having its database moment. Landing in StreamNative Pulsar 5.0, we are making topics work like tables: πŸ“ˆ Scalable Topics: No more guessing partition counts. The system auto-splits/merges range segments based on load while keeping key ordering perfectly intact. 🧠 ORM-Style APIs: 3 purpose-named interfaces (StreamConsumer, QueueConsumer, CheckpointConsumer) turn invalid runtime operations into compile-time errors. πŸ”„ Silent Rebalances: Consumer disconnects hit a grace period first. Rebalances become background infrastructure events, not application fire drills. Streaming infrastructure shouldn't demand so much of its users. Catch up on the full paradigm shift. πŸ‘‡ Link to the blog in the replies!
1
135
🎀 CFP is still open for Data Streaming Summit 2026! πŸ“… Oct 7–8 πŸ“ San Francisco This year, #DSS becomes The Data Streaming Agent Infra Conference β€” bringing together engineers building the infrastructure behind real-time data and AI-native applications. We're looking for talks on: ⚑ Streaming systems at scale πŸ€– Production agent platforms πŸ” Observability, governance & runtime infrastructure πŸŒ‰ Architectures connecting streaming and agent workloads Selected speakers receive a free conference pass and promotion across DSS channels. Submit your talk today! πŸ‘‰ hubs.ly/Q04l2Gxh0 #DataStreaming #ApacheKafka #ApachePulsar #Flink #AIAgents #DSS26
35
Tired of building complex ingestion pipelines just to get your streaming data ready for analytics and AI? Join @streamnativeio and @starburstdata for a live webinar on June 25th to learn how to bridge the gap between real-time Kafka streams and Apache Iceberg tables! 🌊🧊 We will show you how to simplify your journey to actionable insights using an open, modern data architecture. What we’ll cover: πŸ”Ή Pipeline-free Kafka to Iceberg ingestion πŸ”Ή Schema governance best practices πŸ”Ή Optimizing Iceberg tables via LakeOps πŸ”Ή Scaling with open standards (Kafka, Iceberg, Trino) πŸ”— Save your seat: hubs.ly/Q04kT0CT0 #DataStreaming #ApacheKafka #ApacheIceberg #Lakehouse #RealTimeAnalytics #AIInfrastructure
58
StreamNative retweeted
Jun 9
Last week Snowflake launched Datastream β€” Kafka-compatible streaming built directly into the platform. To me that's a strong signal: the industry now agrees streaming and the lakehouse belong together, especially for AI agents that need to act on fresh data. That convergence is exactly the shift we've been building toward. We call it - #Lakestream. AI agents can't act on yesterday's data. They need the latest events β€” seconds old β€” sitting next to the full history that gives them meaning, governed enough to trust. That's real-time context, and it's what the classic two-system setup β€” streaming on one side, the lakehouse on the other, pipelines in between β€” was never designed to deliver. A Lakestream unifies the two, end to end on open standards. You produce and consume the live stream through the Kafka API, and you query that same data as an open Apache Iceberg table from any engine. One stream, written once β€” a low-latency event stream and an open table at the same time. The log that backs the stream is the table, one copy serving the present and the past at once. The lakehouse merged the lake and the warehouse. The #Lakestream does it one step earlier in the data's life: it merges the stream and the lakehouse β€” and keeps both sides open. We wrote up the full thesis πŸ‘‡ streamnative.io/blog/what-is…
1
2
97
What exactly is a #Lakestream? 🌊 It’s a new data streaming architecture designed to solve a massive engineering problem: the costly, fragile separation between real-time streaming logs and analytical data lakes. A Lakestream unifies them into a single system based on 3 pillars: πŸ”„ Stream-Table Duality: One data copy, accessible as a real-time message log or an analytical table. πŸ”“ Open Formats: Data lands directly on cloud object storage using open standards like Apache Iceberg. ⚑️ Zero-ETL: No external connectors, batch sync pipelines, or data duplication required. Read the foundational guide to see how it simplifies modern data infrastructure. πŸ‘‡ Link to the blog in the replies!
1
1
146
Why does traditional Kafka fracture at scale? Because of the leader-follower storage model. Binding partitions to local broker disks leads to massive over-provisioning, ISR rebalancing storms, and brutal cross-AZ replication bills. Ursa For Kafka (#UFK) replaces this with a Leaderless Storage Architecture: 🧠 Decoupled Engine: Compute handles the protocol; storage handles the data. πŸ™…β€β™‚οΈ No Storage Leaders: Data replicates concurrently and directly to cloud object storage. πŸŒͺ️ Zero ISR Storms: No local broker disks means zero data re-syncing when nodes fail. Catch up on the technical blueprint behind our VLDB award-winning engine, #Ursa. πŸ‘‡ Link in the replies! #ApacheKafka #DataEngineering #DataStreaming #StreamNative #Ursa #DataArchitecture
1
117
If you're building real-time applications with @startreedata (#ApachePinot), you need a streaming foundation that scales efficiently. StreamNative Kafka (#UFK) StarTree Cloud delivers: ⚑️ Native Kafka API: Zero code rewrites for your existing apps. ⏱️ Sub-Second Latency: Ultra-fast analytical queries over high-throughput streams. πŸ“‰ 95% Infrasctructure Savings: Powered by our diskless #Ursa engine to eliminate broker storage tax. πŸ”Œ No Connectors: Brilliantly simple, direct ingestion. Catch up on the technical breakdown of the ultimate real-time data stack. πŸ‘‡ Link to the blog in the replies! #RealTimeData #Kafka #Streaming #Analytics #DataEngineering
2
1
136
Kafka Streams ➑️ BigLake Metastore ➑️ BigQuery. Natively. ⚑️ On GCP? Skip brittle sinks and expensive pipelines. With StreamNative Kafka Service – Ursa For Kafka (#UFK), your Kafka topics land as open Apache Iceberg tables in your own GCS bucketβ€”with 0 code changes. πŸ”Ή BigLake-native governance πŸ”Ή Up to 95% infrastructure savings πŸ”Ή Query instantly in BigQuery or Spark Stream once. Own your data. Query anywhere on Google Cloud. πŸ”— Architectural breakdown in the replies. #RealTimeData #Lakehouse #Kafka #BigLake #GoogleCloud #DataEngineering #Streaming
1
114
BIG NEWS: Today we’re announcing our native integration with Snowflake Horizon Catalog! β„οΈπŸš€ Real-time streaming is nothing without real-time governance. Now, you can extend Snowflake’s premier security and compliance perimeter straight to your Kafka streams. StreamNative Kafka Service (#UFK) #Snowflake Horizon Catalog delivers: ⚑️ Zero-ETL: Kafka topics automatically land as open Apache Iceberg tables via #Ursa. πŸ›‘οΈ Automated Cataloging: Streaming metadata automatically registers with Snowflake Horizon. πŸ” Unified Governance: Instantly inherits Snowflake RBAC, data masking, and lineage. πŸ’° 95% Infrasctructure Savings: Cut traditional Kafka broker overhead while keeping data fully governed. Keep your existing Kafka apps. Keep your Snowflake governance. Ditch the complex ETL pipelines. πŸ‘‰ Read the official launch blog: hubs.ly/Q04kgmFb0 #Snowflake #Datastream #Lakestream #Kafka #Horizoncatalog
51
Real-time streaming 🀝 Open data governance. If you use Databricks #UnityCatalog, you don't need expensive data duplication or brittle connectors to feed your managed tables. Ursa For Kafka (UFK) writes native Kafka streams straight into your storage as open Delta/Iceberg formats. ⚑ Kafka API on top ⚑ Unity Catalog underneath ⚑ 0 rewritten apps Catch up on the technical architectural breakdown. Link in the replies!πŸ‘‡ #RealTimeData #Lakehouse #ApacheIceberg #Databricks #Lakestream #UnityCatalog #Kafka
1
107
What if your Kafka topics were your Iceberg tables? No Kafka Connect. No ETL pipelines. No data duplication. Ursa For Kafka (#UFK) runs the native Apache Kafka wire protocol, but writes data directly to your cloud object storage as open lakehouse tables! βœ… Zero Code Changes: Keep your existing Kafka apps. βœ… Open Formats: Data lands as Apache Iceberg or Delta Lake. βœ… 95% Cost Reduction: Diskless architecture eliminates expensive cross-AZ replication. πŸ‘‡ Read the architectural deep dive in the comments! #ApacheKafka #Lakestream #Lakehouse #Iceberg #RealTimeAnalytics
1
127
At #SnowflakeSummit '26, @Snowflake announced Datastream to collapse Kafka streams into Iceberg tables. It completely validates the vision: the divide between streaming and the lakehouse is dead. But while Snowflake is building this specifically to pull workloads into their ecosystem, we built #Lakestream to give you that exact same zero-ETL futureβ€”with total architectural freedom and unprecedented cost savings. While the industry gets on a waiting list, you can ship this week with Lakestream: πŸš€ Production-Ready Zero-ETL: Kafka API on top, Iceberg underneath. πŸ”“ Zero Vendor Lock-in: Data lives in open formats in your storage, queryable by any engine. 🧠 Proven Tech: Powered by #Ursa, winner of the VLDB 2025 Best Industry Paper award. πŸ’° Up to 95% Infrastructure Savings: Diskless, leaderless architecture. Why wait for a preview when you can build the future right now? Stream once. Own your data. Query anywhere. πŸ‘‡ Links to get started in the replies! #DataEngineering #StreamingData #Lakehouse #ApacheKafka #Iceberg #Lakestream
1
128