⏰ Final Reminder – Delta Lake Webinar Tomorrow! Wondering if data engineering design patterns can unlock new insights into Delta Lake? Or how Delta Lake can become a key part of your streaming data architecture? Join @newfront (@bufbuild) and @waitingforcode as they tackle these questions head-on! 🗓️ Oct 14 @ 9AM PT 🎥 Live on LinkedIn, YouTube & X 📍 Reserve your spot today: luma.com/delta-1014 #opensource #oss #deltalake #streaming #dataengineering

Why don’t Iceberg or Delta Lake have secondary indexes? Because analytics workloads and OLTP workloads optimize for opposite I/O patterns. See my dive into data layout, pruning, and what “indexing” really means in open table formats: jack-vanlightly.com/blog/202…

Are you wondering if general concepts like data engineering design patterns can help you learn about #DeltaLake? Or, if it's possible to leverage Delta Lake within your streaming data architecture? In this webinar, Scott Haines and Bartosz Konieczny will answer these two questions. Scott, who gained streaming expertise at Yahoo, Twilio, and Nike, will share with you best practices for leveraging Delta Lake as a component of your streaming architecture. ✅ Bartosz, who recently published Data Engineering Design Patterns, will reverse-engineer a few of these design patterns to explain which Delta Lake features make everything tick. 🗓️ Tuesday, Oct 14 🕝 9AM PT Don't miss it! 🔗 Register today: luma.com/delta-1014 #opensource #oss #dataarchitecture #dataengineering @waitingforcode

Releasing Soon! Pre-order now shroffpublishers.com/books/9… Data Engineering Design Patterns By Bartosz Konieczny  @waitingforcode. with @OReillyMedia Focusing on various aspects of data engineering, including data ingestion, data quality, idempotency, and more. #dataengineering

If you want to understand the consistency models of the mentioned table formats of the paper, I've written about it extensively and written formal models. * jack-vanlightly.com/analyses… * jack-vanlightly.com/analyses… * jack-vanlightly.com/analyses… * github.com/Vanlightly/table-…

Data Engineering patterns on the cloud by Bartosz Konieczny is on sale on Leanpub! Its suggested price is $39.00; get it for $24.65 with this coupon: leanpub.com/sh/ygsnqbRD @waitingforcode #CloudComputing #AmazonWebServices #GoogleCloudPlatform #MicrosoftAzure

Join @newfront and @waitingforcode and learn all about streaming Delta Lake tables with Apache Spark Structured Streaming! 🦀 🗓 March 21st 🕝 9:00AM PT / 12:00PM ET 💻 Join this webinar via LinkedIn, YouTube, or Zoom! Learn more: linkedin.com/events/streamin… #deltalake #streaming

I have been busy the last few months writing a book for O'Reilly about how to build ML systems (batch, real-time, and LLMs), distilling much of what I have learnt from both working with customers as well as students. Why could the book interest you? * Data Scientists - transition from training models to building ML systems * ML Engineers - learn about how to build batch, real-time, LLM systems in modular parts that you compose into a ML system * Data Engineers - learn about the data transformation taxonomy for ML and how badly structured DAGs prevent reuse in ML systems * Architects - divide et impera - learn how modularity helps you build faster and better ML systems. Early access to the first chapter (52 pages) is available here: hopsworks.ai/lp/oreilly-book…

I don't want to start a flame war here, but IMO it is a mistake to jump straight to distributed databases (and 90% of the content below is distributed databases) without first learning fundamentals on single node databases. Here's my 10 things to understand about databases: 1. Relational model. Primary keys, foreign keys, normal form. 2. SQL language. Ideally with advanced SQL (CTE, analytics) 3. ACID and how transactions work 4. Write-ahead log (or binlog) and how it is used. Especially around restarts, recovery and replication. 5. Buffer cache, disk storage layout and how they interact 6. What happens when databases start? when they shut down? 7. Indexes, cluster tables, partitions and other types of database structures. 8. Query parsing, planning and optimizing. 9. MVCC and how to deal with its quirks in your DB of choice 10. Security - authentication, authorization, encryption on wire and at rest. 11. (Bonus) Investigating performance issues and making sense of benchmarks. Entire world, stuff that 99% of developers use daily. You can be a deep expert without ever looking at distributed databases. And this also serves as strong foundation once you do. And if you use Postgres, I found this free book super helpful in making sense of things: interdb.jp/pg/

Data Engineering patterns on the cloud by Bartosz Konieczny is on sale on Leanpub! Its suggested price is $39.00; get it for $26.10 with this coupon: leanpub.com/sh/1T4q5Z81 @waitingforcode #CloudComputing #AmazonWebServices #GoogleCloudPlatform #MicrosoftAzure

Chapter 4 of The Architecture of Serverless Data Systems: CockroachDB (serverless). jack-vanlightly.com/analyses…

The early release of Delta Lake: The Definitive Guide is here! 🎉 The latest edition includes the addition of Chapter 12: Performance Tuning. Download here ➡️ bit.ly/472DVY7 Authors @dennylee, Prashanth Babu, Tristen Wentling, & @newfront #opensource #deltalake #oss

Data Engineering patterns on the cloud: How to solve common data engineering problems with cloud services? leanpub.com/data-engineering… by Bartosz Konieczny is the featured book on the Leanpub homepage! leanpub.com @waitingforcode #CloudComputing #AmazonWebServices

Last week I spent some time to understand the #PySpark applyInPandasWithState. This week I'm refactoring the code, hoping to still understand it 2 months later ;) 👉 waitingforcode.com/apache-sp…

In the previous release #PySpark has got an interesting streaming feature -> the arbitrary stateful processing. It has a different API than the Scala version but is more adapted to the Python world. More 👉 waitingforcode.com/apache-sp…

A list of articles I share again and again when developers ask me about Kafka 🧵

[ANNOUNCEMENT] Congrats to the Apache Spark community and all the contributors! The Apache Spark 3.5.0 release is here. Try it out! spark.apache.org/releases/sp…

It's not a rebranding but more a regrouping 😉 All my additional #dataengineering content is now available from there waitingforcode.com/better (planning to add some stream processing materials soon)

If Delta Lake implemented the commits only, I could stop exploring this transactional part after the previous article. But as for RDBMS, #DeltaLake implements other ACID-related concepts, such as isolation levels 👉 waitingforcode.com/delta-lak…

One of the great features of table file formats is the ability to handle write conflicts. It wouldn't be possible without commits that are the topic of my #DeltaLake blog post. waitingforcode.com/delta-lak…