What happens when Materialize R&D gets a day and a half to build whatever they want?
🔹WASM UDFs
🔹 S3-backed upsert
🔹 Formal verification
🔹 EXPLAIN ANALYZE (now live!)
All from our hackathon 🔗 bit.ly/4oGdXTu#streamingSQL#databases#rustlang
👏Just wrapped up the very first #DolphinDB Tech Deep Dive livestream! We spent an hour diving deep into streaming SQL — the tech that's making real-time, high-frequency data processing actually possible.
For everyone who joined live (and those catching up later), here's what we explored:
⚡ Our recipe for speed:
→ Process only new data (skip costly full scans)
→ Keep data hot in memory (no I/O bottlenecks)
→ Supercharge queries with hybrid indexing
✅ The numbers? We're talking 1,000 mixed operations per second with just 7ms from start to finish. That's the kind of speed that lets companies react to market changes, sensor data, or customer behavior in real-time.
👍Huge thanks to everyone who joined and brought such sharp questions — the Q&A was a highlight.
🔜 Next session: JIT-accelerated computation — you’ll want this on your calendar.
🌐 More info: dolphindb.com/#RealTimeComputing#StreamingSQL#BigData#TechTalk#DataEngineering#JIT#FinTech#IoT#SmartManufacturing
💭DolphinDB Tech Deep Dive | #1: Streaming SQL Processing
We’re excited to kick off our new “#DolphinDB Tech Deep Dive” series — a three-part tech talk designed to unpack the core technologies behind DolphinDB, built for high-performance real-time computing at scale.
In this series, we’ll explore:
1️⃣ Streaming SQL Processing
2️⃣ Just-In-Time (JIT) Compilation
3️⃣ GPU Acceleration
🔹 Episode 1 Details
🗓 Date: Thursday, August 7 | 7:30 PM (GMT 8)
🎙 Speaker: Dr. Xuntao Cheng, Deputy Director of R&D
Don't miss this opportunity to hear directly from our engineering team about the technical innovations driving real-time data processing.
📩 Register via email: info@dolphindb.com
🌐 More info: dolphindb.com/#RealTimeComputing#StreamingSQL#BigData#TechTalk#DataEngineering#JIT#GPU#FinTech#IoT#SmartManufacturing
🔍 Diving Deeper into Caching & Computation Graphs! 🚀
When dealing with intricate computation graphs, the key lies in:
Granular Change Tracking: Using CDC, we can capture not just that a change occurred, but specifically what changed. This granularity is pivotal for dependency management.
Real-time Dependency Management with Streaming SQL: Modern streaming SQL platforms allow for on-the-fly data processing. As CDC captures changes, they're fed into these platforms. By implementing logic within our streaming SQL, we can track which parts of our computation graph are affected by a change.
Scoped Memoization: Store intermediate computations and, when a change occurs, recompute only the affected parts of the graph. Efficiency at its best!
In essence, combining CDC's detailed change tracking with the real-time processing capabilities of streaming SQL provides a robust solution to the challenges of caching in complex computation environments.
#Caching#CDC#StreamingSQL#DataManagement
1/ We're back with a new post in our #streamingSQL series, this time exploring the Sliding Window Hash Join (SWHJ) algorithm. Shout out to @ApacheArrow and #DataFusion for providing us a great foundation on which we built our implementation.
synnada.medium.com/the-slidi…
In this article, we compare running the same job in Flink's Datastream and Table APIs and discuss how the job's performance differs depending on the chosen API.
Follow us in this journey of beating #FlinkSQL's performance 👇
bit.ly/3wfwL2a#SQL#streamingSQL
This articles showcases how you can build real-time applications with Flink #SQL and the use of MATCH_RECOGNIZE without touching a line of Java or Scala code
Find out more 👇
bit.ly/3lbSPVB#ApacheFlink#FlinkSQL#streamingSQL