Wir waren die ersten in Europa, die eine objektorientierte Datenbank (ObjectStore) eingesetzt haben. Nach einem Jahr Evaluierung durch mehrere Forschungszentren von ABB. Unschlagbar schnell. Wir haben die später wieder rausgenommen, weil die OEMs keine neuen Datenbanken wollten. Man integriert einfach Engineering Objects auf einem gängigen PDM-System, das als Engineering Data Backbone fungiert.

I've been thinking about this as well. Because this is slatedb based, it's naturally embedded. So the program can just link the DB into itself, with all state being durable on objectstore at all times. You don't need the a namespace at all.

Agreed. If the end state is to dump it in the warehouse anyway, doing the enrichment there is a whole lot more efficient, which is what Buffer enables. Re: losing the current batch, that is indeed a risk, especially since you have to batch more to make the objectstore costs palatable. But Kafka also has the same problem to a smaller extent. If the root producers are set up for retries, then there is no fundamental problem.

Well, my design was based on optimistic locking (which came from the already mentioned ObjectStore), so your application had to be able to handle a failed transaction. But I suspect that the difference is that in my design there was a clean committed/failed feedback, not just “lost somewhere”.

Polars愛用しているけど、GCPの Workload Identity Federation github.com/pola-rs/polars/is… が使えない問題（Polarsに限らずrustのObjectStoreに依存している奴は全部ダメ）は本当に困っている。解決される見込みなさそうだよなぁ。。。

✅javascript 学習：1h30分 今年の家計簿整理と来年予算の家族会議も完了 ・objectStoreにcreateIndexを作ると、titleなどのプロパティ検索が可能（upgradeneededのみ） ・unique：重複不可 ・multiEntry：配列を分解して検索対象にできる

✅筋トレ ✅javascript 学習：15分 ・トランザクション：全部成功or全部失敗 ・途中エラーやabort：処理はなかったことに ・objectStoreの操作：トランザクション内でまとめて実行 ・complete：全処理が成功したときのみ発火 ・エラー発生や transaction.abort()でabortが発火 #デイトラコミュニティ

I think objectstore-only breaks down if you need low-latency ingest, read-your-write semantics, or have entities that are frequently updated. I'm not sure if there is a generalizable layer in between that can bridge those gaps. It's omething we've been thinking about lately!

90年代後半にObjectStoreとかObjectivityとかのグラフDB／オブジェクトDBが出てきた時はおおっと思ってJavaWorld誌に長い特集記事書いたりしたけど、結局扱いの難しさやメンテコストの高さで流行らなかった。数10個のLEO衛星群のトポロジー管理に実用してた例はかっこよかったな。

Toda empresa séria, obrigatóriamente, tem BACKUPS in-loco e externos TESTADOS para rápido recovery. Em matéria de backups você tem dois tipos: 1. Volume Snapshots (local) -> primeira linha de recovery, são rapidos. 2. ObjectStore (GCP, AWS - externo) -> Disaster Recovery.

DAOS and the Ups and Downs of SDS (Software-Defined Storage). @HartmutWiehr @manageITmagazin bit.ly/46slVsA #DAOS #MultiCloud #HPCStorage #HighPerformanceStorage #ObjectStore #HPC #AI #ITPT @ITPressTour 61st Edition in London

SQL was there before you were born and SQL will be there after you die: Somebody invents a "SQL replacement" every decade. It then fails, gets ignored, or worse, its good bits get unceremoniously absorbed into the SQL standard, leaving the original "revolutionary" system to wither on the vine, much like Sisyphus is doomed to push his boulder. Remember the NoSQL craze of the late 2000s? "SQL is dead! Documents are the future!" Fast forward a bit, and what do we see? MongoDB, the poster child of that movement, now has a SQL interface. Cassandra has CQL. The supposed usurpers have, in many ways, bent the knee to the established king. The funny thing is, while these "next big things" often flame out, the old stuff? It sticks around. There are IBM IMS databases, relics from the 1960s Apollo moon missions, still chugging away in major banks, processing your ATM transactions. Why? Because ripping out a core system that (mostly) works is like performing open-heart surgery with a rusty spork – risky and expensive. And it’s not just the systems; the ideas from these ancient behemoths are still surprisingly relevant. Think query compilation – turning your SQL into efficient machine code. IBM was doing that in assembly in the 1970s with System R. We use LLVM now, but the core challenge? Still the same. This is why a stroll through database history isn't just an academic exercise; it’s a critical lesson in not reinventing the flat tire. If we don't learn from the ghosts of databases past, we're just setting ourselves up to repeat their costly mistakes. So, let's take a quick, unfiltered dive into how we got here. The Pre-Relational Dark Ages: Pointers, Hierarchies, and Programmer Pain (1960s) Imagine it’s the 1960s. You’re NASA, literally trying to shoot for the moon. You have an astronomical number of parts, suppliers, and intricate dependencies. Your data is a tangled mess. Enter systems like IDS (Integrated Data Store). GE originally cooked this up for a timber company with a massive inventory problem. Think of your data as a giant spiderweb, where every piece of information was connected to others by physical pointers on disk or in memory. If one of those pointers got corrupted? Your entire database was toast. Programmers, armed with languages like COBOL, had to write complex, nested loops to navigate this maze, one agonizing record (or "tuple") at a time. In a classic corporate blunder, GE, deciding they weren't #1 in the computer biz, sold off their entire computing division to Honeywell. So much for that moonshot. Then there was IMS (Information Management System) from IBM, the powerhouse behind the actual Apollo program's parts tracking. IMS went for a hierarchical model – think of a rigid, top-down organizational chart for your data. A part could only be supplied by one vendor in its defined hierarchy. If "Battery Model Z" was available from three different suppliers, you had three complete, redundant copies of "Battery Model Z's" details. Need to update the battery's specs? Good luck hunting down every single instance. Even worse, how the data was physically stored – as a hash table or a B-tree – was hardcoded into the application. If you realized a B-tree would be better for range queries after initially choosing a hash table, you weren't just changing a setting; you were dumping data, reloading it, and rewriting your application code because the API itself changed. Yet, this beast, or its descendants, still processes your banking transactions today. The lesson? Inertia is a powerful force. Around the same time, Charles Bachman, who'd worked on IDS, pushed for a standard way for COBOL programmers to talk to databases. This led to CODASYL (Conference on Data Systems Languages), which championed the "network data model." It was like IDS on steroids – more pointers, more sets, more ways for your data to become an unmanageable spaghetti monster. Bachman even won a Turing Award for this. The Enlightenment: Ted Codd & The Relational Revolution (1970s) Working at IBM, a mathematician named Ted Codd saw the IMS programmers tearing their hair out. He realized the insanity of coupling the logical view of data with its physical storage and the inefficiency of tuple-at-a-time processing. His 1970 paper proposed a radical new approach: Simple Data Structures: Store data in simple tables (relations). High-Level Language: Let users declare what data they want, not how to get it, step-by-step. This crucial insight sparked the development of powerful new query languages. At UC Berkeley, the Ingres project, led by Michael Stonebraker, developed QUEL (QUEry Language) around 1974, directly based on Codd's relational algebra. Concurrently, at IBM, researchers on the System R project were developing SQL (Structured Query Language), with its initial designs also appearing in the mid-1970s. Physical Independence: Separate the logical data view from the physical storage. Let the database figure out the best way to store and retrieve. When Codd published his paper in 1970, the old guard scoffed. 'A machine write queries better than a human? Preposterous!' Yet, both QUEL and SQL aimed to do just that, providing high-level ways to interact with relational data. While Ingres with QUEL gained early traction in academic and research circles, and Stonebraker still insists QUEL was superior, it was SQL that eventually won the broader industry adoption. This was partly due to IBM's significant market influence when they later commercialized SQL with DB2 in the early 1980s, effectively making it the de facto standard. The path to SQL's dominance also involved the famous story of Larry Ellison at Oracle closely following and implementing System R's ideas. The standards bodies, too, ultimately leaned towards SQL, a decision Stonebraker famously attributes to his own disdain for such committees, leading him not to push QUEL as aggressively in those forums. Meanwhile, a sharp character named Larry Ellison saw an opportunity. Legend has it he’d get IBM’s research papers (sometimes by just calling up researchers who were happy to share their "academic" work) and implement the ideas in his own fledgling system: Oracle. When IBM finally launched DB2 in the early 1980s, it was a signal: the relational model was here to stay. And because IBM chose SQL (originally SEQUEL – Structured English QUEry Language – later shortened to SQL due to a trademark dispute) as its language, SQL became the de facto standard. Oracle, having already bet on SQL, was perfectly positioned to ride the wave. Ingres, with QUEL, eventually added SQL support, but by then, the race was largely decided. The "SQL is Dead" Cycles: A Recurring Theme The 1980s solidified relational dominance, but the "SQL is flawed/dead" narrative was just getting started. Object-Oriented Databases (Late 1980s/Early 1990s): C was hot. Programmers grumbled about the "impedance mismatch" – relational tables didn't map cleanly to their beloved objects. "Why can't we just store objects directly?" they asked. Thus, OODBs like ObjectStore and Versant were born. Their Achilles' heel? No standard query language (OQL arrived too late and never caught on), and applications were tightly coupled to specific OODB APIs. The good ideas (like richer data types) were eventually, you guessed it, absorbed into the relational model, leading to "object-relational" systems like PostgreSQL (which, amusingly, was initially developed in LISP by Stonebraker's team post-Ingres). The "Boring" 90s & The Internet Explosion (2000s): The 90s were a period of refinement. Microsoft forked Sybase to create SQL Server. A Finn named Michael "Monty" Widenius created MySQL (My was his daughter's name; he later created MariaDB for his other daughter, Maria, and MaxDB for his son Max). PostgreSQL, having shed its LISP and QUEL origins, embraced SQL. Then the internet hit, and suddenly, even small outfits could generate (and drown in) massive datasets. The Rise of Analytical Databases & MapReduce: People weren't just transacting; they wanted to analyze their growing data piles. Row-oriented databases choked on analytical queries. Early attempts like "data cubes" (pre-computed aggregations) were a stopgap. Then came specialized analytical databases, many of them forks of PostgreSQL, pioneering columnar storage (Netezza, Vertica, Greenplum). This was a game-changer for analytics. Simultaneously, Google, needing to process its colossal web crawl, invented MapReduce. Yahoo! quickly cloned it as Hadoop. Instead of SQL, you wrote custom Java functions for map and reduce stages. Programmers were back to defining data parsing logic from scratch, a huge step backward in terms of abstraction. While initially hyped, the inefficiencies of MapReduce for general-purpose analytics became apparent, and SQL layers (like Hive) were awkwardly bolted on top. NoSQL (The Big One - 2000s): The internet scale also revived the "SQL is too slow/rigid" argument with unprecedented force. The NoSQL movement (MongoDB, Cassandra, DynamoDB) championed "schema-less" designs, horizontal scalability, and often, "eventual consistency" over strict ACID transactions. "We need to be always on, even if the data is a bit weird sometimes!" was the mantra. The irony, as we've seen, is that most of these systems have since added SQL-like interfaces and stronger consistency options. The Modern Whirlwind: Clouds, Lakes, and More "Revolutions" (2010s - Present) The cycles continue, now accelerated by the cloud: NewSQL & Distributed SQL: The goal was NoSQL's scalability but with SQL and ACID compliance. Early NewSQL systems had mixed success. Their spiritual successors, often branded "Distributed SQL" (CockroachDB, TiDB, YugabyteDB), are learning from past mistakes and gaining more serious traction. The Cloud Transformation & Data Lakes: Cloud platforms (AWS, Azure, GCP) fundamentally changed database architecture. The dominant model shifted from shared-nothing to shared-disk (think Snowflake), where compute and storage are separated, often using object stores like S3. This gave rise to Data Lakes, where raw data in open formats (Parquet, ORC) is dumped into cheap cloud storage, accessible by various query engines. Graph Databases: "Your data is inherently connected, so store it as a native graph!" (Neo4j, TigerGraph). It’s the OODB argument in a new hat. Unsurprisingly, SQL is now incorporating graph query capabilities (SQL/PGQ). Recent research even shows general-purpose analytical databases like DuckDB, with graph-specific optimizations (like advanced join algorithms), can outperform specialized graph systems. My bet? Graph databases will be a valuable niche, and the relational model will swallow the best ideas. Time Series Databases: A specialized flavor of relational, optimized for the firehose of data from IoT devices, application metrics, and financial tickers (TimescaleDB, InfluxDB). They handle time-windowed queries and high-throughput append-only workloads very well. Vector Databases (The Current Darling): Fueled by the AI/ML boom, these are designed to store and search high-dimensional vector embeddings (the "thoughts" of AI models). Most are essentially document databases with an Approximate Nearest Neighbor (ANN) index grafted on. Guess what? Mainstream relational databases are already adding vector data types and ANN indexing capabilities. This will likely become just another feature in the ever-expanding relational toolkit. Blockchain Databases (The Emperor's New Clothes): "Decentralized! Trustless! Immutable Ledger!" The pitch is seductive. The reality, for most database use cases, is a performance nightmare and a solution desperately searching for a problem that a traditional database with proper auditing and access control can't solve more efficiently and cheaply. Amazon even built QLDB, which offers the immutable ledger part without the decentralized blockchain overhead, implicitly acknowledging where the real (limited) value lies. Still Spinning: The Enduring Power of Relational So, after decades of supposed revolutions, where are we? The relational model, with SQL as its lingua franca, isn't just surviving; it's thriving. It's a cockroach in the best sense – adaptable, resilient, and capable of absorbing almost anything thrown at it. The core principles Ted Codd laid out – data independence, simple structures, a high-level way to ask for what you want – are timeless. The next time a charismatic founder on a conference stage tells you their shiny new database will make SQL obsolete, remember this journey. History doesn't just repeat itself in the database world; it often comes around with a new marketing budget and a fresh set of buzzwords, only to find the old ideas were pretty good after all.

A very interesting session about #DAOS led by J. Lombardi during the last edition of The @ITPressTour in London #ObjectStore #HPC #AI #AccessMethods #POSIX #NAS #S3 #Hadoop #U3 #Flash #SSD #SDS #NVMe #FastIO #ErasureCoding #ITPT pn7.fr/0B72BE40

Local players are changing the data storage landscape. @crn_polska bit.ly/44bd0uq #Auwau #Daos #Storadera #MultiCloud #CloudBackup #Backup #DataProtection #SaaS #HPCStorage #ObjectStore #HPC #AI #SecondaryStorage #CloudStorage #ObjectStorage #S3 #ITPT @ITPressTour

Day 2 of The @ITPressTour w/ #Auwau #DAOS #Storadera #MultiCloud #CloudBackup #Backup #DataProtection #SaaS #HPCStorage #HighPerformanceStorage #ObjectStore #HPC #AI #SecondaryStorage #CloudStorage #ObjectStorage #S3 #ITPT

@abperiasamy of @Minio is our biggest source of knowledge for storage technology of the day. @garimakap @mohamedimran_kr #OpenSource #ObjectStore #ExaScale #GTC2025

Some really good practical examples here -> Watch: We need to talk about #Microservices... And how #NATSio (@NATS_io) can simplify things -> youtu.be/AiUazlrtgyU?feature… via @Synadia #servicediscovery #dataconnectivity #opensource #cncf #platformengineering #devops #datastreaming #objectstore #keyvaluestore #edge #ai #edgeai #ml #machinelearning #edgecomputing #multicloud #iot #iiot

Outside of @awscloud, no player has done more to spread the gospel of #S3 than MinIO. Come meet the founders behind the most broadly deployed #objectstore in the private cloud, and learn how we are revolutionizing large-scale data infrastructure—across clouds, CPUs and GPUs. You can find us at #AWSreInvent 2024 (booth 576) or by booking time with our founders here: minio.chilipiper.com/me/manj…

В общем, это проект по интеграции кучи внешних и легаси-внутренних источников данных в одну DDD систему, и создания одного HA source of truth в лице постгрес-бд. Основной сложностью было свести ±стандартов в один, и написать новые либы для ботлнеков. Я не буду очень сильно вдаваться в детали, но: NAST Jetstream NATS Object Store работают хорошо, Spring Boot микросервисы скалируются хорошо, постгрес с timescale хорошо держит ~20тб данных в этом юзкейсе, больше нам и не нужно. Команды которые не умеют писать тесты – не заебись. Люди, которые не умеют нормально писать на спринге – производят очень печальный код. Дальше у нас перевод всей MQ коммуникации с Active MQ на NATS. Timescale работает стабильно, но про Foreign Key's с ним можно забыть. NATS'овский протобаф можно отлично использовать в джаве как модель. Кубер отличная вещь, но нужны люди в каждой команде, которые будут прямо целенаправленно заниматься инфраструктурой. Лучшая комбинация скорости и надёжности оказалась в заказе данных ивентами в JetStream и передача крупных (100мб-10гб) пэйлоадов в ObjectStore. Чистая MQ передача пэйлоада с определённого момента ломается. С кафкой или NATS получается готовить HA решения, при этом кафка неудобна. Джава может хорошо в быстрый IO, лучше чем комбинация JNA и плюсовых либ. Как-то так

We are excited to announce our new optimizations and benchmarks for @Arm-based chipsets powering our #objectstore. These optimizations underscore the relevance of power-efficient, computationally dense chips in key AI-related tasks and demonstrate the capabilities of the Arm architecture for modern #AI and data processing workloads including erasure coding, bit rot protection and encryption. To learn more about the benchmark results, check out this article detailing the optimizations and testing: blog.min.io/bringing-arm-int…