Unity Catalog is the industryโ€™s only universal catalog for data and AI.

Joined July 2024
172 Photos and videos
[Ep 1] Open Lakehouse AI: The Catalog Layer, Interoperability & AI-Agent Governance Learn how composable lakehouses center the catalog for metadata, versioning, and commit coordination. ๐Ÿ‘‡ ๐Ÿ”ธ Interop: Iceberg REST, Unity Catalog (OSS), Spark, Delta Lake, Delta-RS, Iceberg. ๐Ÿ”ธ Governance: credential vending, row/col filters, column masks, audits, and why โ€œno governance was the easiest governance.โ€ ๐Ÿ”ธ Agents as data customers (Temporal). Fine-grained access beats blanket credentials on 24/7 workloads. ๐ŸŽฅ Full episode: youtube.com/watch?v=dEFAkS7vโ€ฆ #OpenLakehouse #UnityCatalog @ApacheSpark @ApacheIceberg @DeltaLakeOSS
1
2
5
330
Simplified Governance with Catalog-Managed Tables ๐ŸŒ Unity Catalog 0.4.0 introduces support for UC managed tables, enabling data teams to centrally govern, discover, access, and audit their data through Unity Catalog. Instead of relying on scattered storage paths, separate credentials, and manual maintenance, teams can rely on Unity Catalog as the single logical system of record for their data estate. Leverage UC managed tables to strengthen governance, improve performance, and build on the most modern open catalog for the data and AI era. ๐Ÿ“– Check out the announcement and implementation details on the Unity Catalog blog: lnkd.in/eZWkMBaR #unitycatalog #governance #opensource #catalogs #deltalake
1
2
129
With UC managed tables, teams unlock: ๐Ÿ”ธ ๐—จ๐—ป๐—ถ๐—ณ๐—ถ๐—ฒ๐—ฑ ๐—ด๐—ผ๐˜ƒ๐—ฒ๐—ฟ๐—ป๐—ฎ๐—ป๐—ฐ๐—ฒ:ย Unity Catalog centralizes access control, replacing fragmented storage-level policies. This simplifies how teams ensure all engines access data in a governed, consistent manner. ๐Ÿ”ธ ๐—ฆ๐˜๐—ฎ๐—ป๐—ฑ๐—ฎ๐—ฟ๐—ฑ๐—ถ๐˜‡๐—ฒ๐—ฑ ๐—ฑ๐—ถ๐˜€๐—ฐ๐—ผ๐˜ƒ๐—ฒ๐—ฟ๐˜†: Unity Catalog provides stable logical table identifiers, eliminating the need for clients to depend on physical storage paths for discovery. ๐Ÿ”ธ ๐—˜๐—ณ๐—ณ๐—ผ๐—ฟ๐˜๐—น๐—ฒ๐˜€๐˜€ ๐˜๐—ฎ๐—ฏ๐—น๐—ฒ ๐—ผ๐—ฝ๐˜๐—ถ๐—บ๐—ถ๐˜‡๐—ฎ๐˜๐—ถ๐—ผ๐—ป๐˜€: By automating storage tuning and credential management, Unity Catalog removes the burden of manual operational maintenance from data teams. ๐Ÿ”ธ ๐—›๐—ผ๐—น๐—ถ๐˜€๐˜๐—ถ๐—ฐ ๐—ฎ๐˜‚๐—ฑ๐—ถ๐˜๐—ฎ๐—ฏ๐—ถ๐—น๐—ถ๐˜๐˜†:ย Metadata and permissions are centralized in a single interface, allowing for high-level oversight of ownership and access instead of parsing low-level storage logs. ๐Ÿ”ธ ๐—˜๐—ป๐—ณ๐—ผ๐—ฟ๐—ฐ๐—ฒ๐—ฎ๐—ฏ๐—น๐—ฒ ๐—ฐ๐—ผ๐—ป๐˜€๐˜๐—ฟ๐—ฎ๐—ถ๐—ป๐˜๐˜€: Unity Catalog can authoritatively validate or reject schema and constraint changes, preventing incompatible updates that could compromise data integrity or break downstream workloads. ๐Ÿ”ธ ๐—™๐—ฎ๐˜€๐˜๐—ฒ๐—ฟ ๐—พ๐˜‚๐—ฒ๐—ฟ๐˜† ๐—ฝ๐—น๐—ฎ๐—ป๐—ป๐—ถ๐—ป๐—ด ๐—ฎ๐—ป๐—ฑ ๐—ณ๐—ฎ๐˜€๐˜๐—ฒ๐—ฟ ๐˜„๐—ฟ๐—ถ๐˜๐—ฒ๐˜€: Unity Catalog delivers table metadata directly to Delta clients, bypassing cloud storage requests to significantly reduce metadata latency and accelerate query planning and writes.
60
We are excited to announce ๐—จ๐—ป๐—ถ๐˜๐˜† ๐—–๐—ฎ๐˜๐—ฎ๐—น๐—ผ๐—ด ๐Ÿฌ.๐Ÿฐ.๐Ÿฌ which includes exciting new features and many bug-fixes and improvements! ๐ŸŽŠ Check out some of the highlights ๐Ÿ‘‡ ๐—จ๐—– ๐—ฆ๐—ฒ๐—ฟ๐˜ƒ๐—ฒ๐—ฟ โš™๏ธ Storage Credentials for AWS โš™๏ธ External Locations for AWS โš™๏ธ Managed storage location for catalogs and schemas ๐—จ๐—– ๐—ฆ๐—ฝ๐—ฎ๐—ฟ๐—ธ ๐—–๐—ผ๐—ป๐—ป๐—ฒ๐—ฐ๐˜๐—ผ๐—ฟ โšก Credential Renewal Enabled by Default โšก Support for Spark 4.1 and Delta 4.1 โšก Atomic CTAS for Delta Tables in UCSingleCatalog ๐—จ๐—– ๐—”๐—œ ๐Ÿค– DSPY Integration with AI Functions A huge thank you to the awesome community who made this release possible! ๐Ÿ“– Full release notes: github.com/unitycatalog/unitโ€ฆ #unitycatalog #opensource #oss
1
113
Weโ€™re excited to announce the release of Unity Catalog v0.3.1!ย ๐ŸŽ‰ This release includes exciting new features and many bug-fixes and improvements. Version 0.3.1 focuses on three major areas: ๐Ÿ”น ๐—œ๐—บ๐—ฝ๐—ฟ๐—ผ๐˜ƒ๐—ฒ๐—ฑ ๐—๐—ฎ๐˜ƒ๐—ฎ ๐—ฐ๐—น๐—ถ๐—ฒ๐—ป๐˜ ๐—”๐—ฃ๐—œ ๐˜„๐—ถ๐˜๐—ต ๐—ข๐—”๐˜‚๐˜๐—ต ๐˜€๐˜‚๐—ฝ๐—ฝ๐—ผ๐—ฟ๐˜: designed for reliability and extensibility in production environments. ๐Ÿ”น ๐—”๐˜‚๐˜๐—ผ๐—บ๐—ฎ๐˜๐—ถ๐—ฐ ๐—ฐ๐—ฟ๐—ฒ๐—ฑ๐—ฒ๐—ป๐˜๐—ถ๐—ฎ๐—น ๐—ฟ๐—ฒ๐—ป๐—ฒ๐˜„๐—ฎ๐—น: to support long-running workloads cross cloud platforms. ๐Ÿ”น ๐—จ๐—–-๐—บ๐—ฎ๐—ป๐—ฎ๐—ด๐—ฒ๐—ฑ ๐——๐—ฒ๐—น๐˜๐—ฎ ๐˜๐—ฎ๐—ฏ๐—น๐—ฒ๐˜€: this enables Unity Catalog to coordinate table storage and commits centrally. This release is the result of contributions from our growing open-source community. A big thank-you to everyone who reported issues, submitted pull requests, reviewed code, and shared feedback! ๐Ÿ”— Dive into the release notes for the full list of highlights:ย github.com/unitycatalog/unitโ€ฆ #unitycatalog #opensource #oss #catalog
79
Managing data pipelines at scale is complicated, often resulting in the data silo problemโ€”where valuable assets are spread across systems. This makes it difficult to track, secure access, and scale cleanly. The solution is a clear structure paired with centralized governance. โœ… The Medallion Architecture structures your data into three distinct layers: ๐Ÿฅ‰ ๐—•๐—ฟ๐—ผ๐—ป๐˜‡๐—ฒ: Raw, ingested data. ๐Ÿฅˆ ๐—ฆ๐—ถ๐—น๐˜ƒ๐—ฒ๐—ฟ: Cleaned, enriched data. ๐Ÿฅ‡ ๐—š๐—ผ๐—น๐—ฑ: Business-level data, ready for reporting. Pair this framework with Unity Catalog, and you get a unified system to manage, govern, and organize your entire data flow. ๐Ÿ”— Walk through how this works: unitycatalog.io/blogs/buildiโ€ฆ #UnityCatalog #MedallionArchitecture #DataGovernance #OpenSource
1
1
83
โšก๏ธ From vector DB to multimodal lakehouse at petabyte scale. Join us in Mountain View on Nov 13 for โ€œScaling Multimodal AI Lakehouse with Lance & LanceDBโ€ with Chang She (@lancedb) at the Open Lakehouse AI Mini Summit! Learn: ๐Ÿ”น Lowโ€‘latency random access search APIs (vectors, text, binaries) ๐Ÿ”น Schema primitives across blobs metadata for feature engineering ๐Ÿ”น Hybrid search (vector fullโ€‘text) for training/fineโ€‘tuning at scale Two exciting tracks, one epic afternoon. ๐Ÿ‘ #OpenLakehouse #AI #LanceDB #VectorDB #DataEngineering
2
7
450
Unity Catalog retweeted
Join us at Open Lakehouse AI Paris on November 24, 6:30โ€“10PM โ€” co-located with the Forward Data Conference! ๐Ÿ‡ซ๐Ÿ‡ท Secure your spot now โฌ‡๏ธ luma.com/OLM-1124 Weโ€™re bringing together data innovators and open source contributors for an evening packed with insight and inspiration. Hear talks from: โœ… Alexandre BERGERE (@DataGalaxy / Datalex) on Building a Scalable Usage Insights Platform with Delta Sharing โœ… Bartosz Konieczny (@waitingforcode) on Design Patterns for the Open Lakehouse โœ… Youssef Mrini & El Ghali Benchekroun (@Databricks) on The Future of Open Table Formats & @unitycatalog_io Food, drinks, networking โ€” and plenty of new ideas (& swag) to take home. ๐ŸŒŸ #opensource #deltalake #oss #apacheiceberg #unitycatalog #lakehouse #openlakehouse #ai
5
8
679
Missed it liveโ“See how an observabilityโ€‘first Telemetry Lake correlates data movement with system behavior to detect issues, diagnose root causes, and adapt in real timeโ€”why correlation > coverage and why the first ~150 characters of signal matter. Watch: youtube.com/watch?v=r04iCvm6โ€ฆ Whatโ€™s inside: @OpenLineage @opentelemetry an LLM reasoning layer that turns noisy lineage and traces into prioritized actions to cut TTD/TTR and reduce blast radius. #openlakehouse #datalineage #observability #AI
1
2
100
Where does the feature platform fit in a world increasingly dominated by AI? At Open Lakehouse AI Mini Summit, Hao Xu (Apple) shares how Feast is evolving beyond a feature store into a full feature platform for AIโ€”bridging data, models, and applications through innovations like Compute Engine, Feast for RAG, and On-Demand Feature Views. Donโ€™t miss this deep dive into how foundational feature architecture continues to drive real-world AI innovation. ๐Ÿ“ Mountain View, CA ๐Ÿ—“๏ธ Nov 13 ๐Ÿ•ฆ 12:00 - 4:30PM PT ๐Ÿ”— Secure your spot: luma.com/OLMS-1113 #opensource #oss #unitycatalog #openlakehouse #ai
2
3
479
๐Ÿšจ Exciting news: @starburstdata has announced GA support for Unity Catalog, enabling users to read and write to any UC managed table โ€” @DeltaLakeOSS or @ApacheIceberg โ€” using industry-standard open APIs! ๐Ÿš€ What you should know: โœ… ๐—–๐—ฒ๐—ป๐˜๐—ฟ๐—ฎ๐—น๐—ถ๐˜‡๐—ฒ๐—ฑ ๐—ด๐—ผ๐˜ƒ๐—ฒ๐—ฟ๐—ป๐—ฎ๐—ป๐—ฐ๐—ฒ ๐—ฒ๐˜ƒ๐—ฒ๐—ฟ๐˜†๐˜„๐—ต๐—ฒ๐—ฟ๐—ฒ โ€” UC permissions are enforced wherever you query, with Starburst authenticating via OAuth 2.0 for per-user, secure access. โœ… ๐—จ๐—ป๐—ถ๐—ณ๐—ถ๐—ฒ๐—ฑ ๐—ฐ๐—ผ๐—บ๐—บ๐—ถ๐˜ ๐—ฐ๐—ผ๐—ผ๐—ฟ๐—ฑ๐—ถ๐—ป๐—ฎ๐˜๐—ถ๐—ผ๐—ป โ€” UC acts as the single commit coordinator for Delta Lake catalog-managed writes, enabling consistency and multi-table transactions across engines. โœ… ๐—ข๐—ป๐—ฒ ๐—ผ๐—ฝ๐—ฒ๐—ป, ๐—ด๐—ผ๐˜ƒ๐—ฒ๐—ฟ๐—ป๐—ฒ๐—ฑ ๐—น๐—ฎ๐—ธ๐—ฒ๐—ต๐—ผ๐˜‚๐˜€๐—ฒ โ€” A single catalog for multi-engine reads/writes, advanced governance, and open table formats โ€” no silos, no lock-in. The open lakehouse continues to grow through community-driven innovation. With Starburst joining engines and tools like @duckdb, @ClickHouseDB, @anyscalecompute, @langchain, and @daftengine, developers can now operate with consistent governance, broad interoperability, and transparent metadata standards across the entire data and AI lifecycle. ๐Ÿ”— Learn more: starburst.io/blog/starburst-โ€ฆ #opensource #oss #unitycatalog #starburst
2
83
Unity Catalog retweeted
The open source and data community is coming together in Mountain View, CA! ๐Ÿ’ฅ Join us Nov 13 (12โ€“4:30PM PT) for the ๐—ข๐—ฝ๐—ฒ๐—ป ๐—Ÿ๐—ฎ๐—ธ๐—ฒ๐—ต๐—ผ๐˜‚๐˜€๐—ฒ ๐—”๐—œ ๐— ๐—ถ๐—ป๐—ถ ๐—ฆ๐˜‚๐—บ๐—บ๐—ถ๐˜, featuring two tracks packed with insights on AI infrastructure, context engineering, and the future of interoperable data systems. Lunch, swag, and great conversations included! Stick around for the @ApacheSpark Happy Hour (5โ€“6:30PM PT) โ€” the perfect way to wrap up a day of learning and community. ๐Ÿ˜Ž ๐ŸŽŸ๏ธ RSVP here: luma.com/OLMS-1113 #opensource #oss #unitycatalog #apachespark #deltalake #apacheiceberg #ai
3
9
1,030
๐Ÿ“ฃ Get ready for the nextย Open Lakehouse AIย webinar! Joinย @wslulciuc, Co-Founder & CEO ofย @OleanderHQ, hosted byย @lisancao from @databricks, will explore why the future of data lineage isnโ€™t another graphโ€”itโ€™s aย reasoning layerย that connects lineage, telemetry, and AI. Learn how Oleanderโ€™sย Telemetry Lakeย unifies @OpenLineage, @opentelemetry, and #LLM reasoning to help data teams detect, diagnose, and adapt in real timeโ€”building theย always-on-call data engineer. ๐Ÿš€ ๐Ÿ—“๏ธ October 30 ๐Ÿ• 9:00AM PT ๐Ÿ”— Register: luma.com/openlakehouse-1030 #opensource #oss #openlakehouse #ai #oleander
3
192
Unity Catalog retweeted
Open Lakehouse AI spotlight: Hannes Mรผhlheisen shares how anyone can contribute to @duckdb and DuckLakeโ€”both MIT-licensed, open source, and welcoming PRs and community engagement on GitHub. Check out this clip to hear how easy it is to get involved and start contributing today. ๐Ÿš€ Whatโ€™s next?โ€‹ ๐Ÿ”น October 30: Webinar โ€” The Failed Promises of Data Lineage: Why More Metadata Isnโ€™t the Answerโ€‹ ๐Ÿ”น November 13: Mini Summit โ€” Mountain Viewโ€‹ ๐Ÿ”น November 24: Meetup โ€” Parisโ€‹ ๐Ÿ”— RSVP and details:ย luma.com/openlakehouse #opensource #oss #openlakehouse #ai #ducklake #duckdb
1
4
436
Open Lakehouse AI brings builders and practitioners together to share how open lakehouse and AI meet in the real worldโ€”through adoption stories, handsโ€‘on patterns, and collaborative learning. ๐ŸŒŽ Join a global community shaping the future of data and AI! Whatโ€™s next? ๐Ÿ”น ๐—ข๐—ฐ๐˜๐—ผ๐—ฏ๐—ฒ๐—ฟ ๐Ÿฏ๐Ÿฌ: Webinar โ€” The Failed Promises of Data Lineage: Why More Metadata Isnโ€™t the Answer ๐Ÿ”น ๐—ก๐—ผ๐˜ƒ๐—ฒ๐—บ๐—ฏ๐—ฒ๐—ฟ ๐Ÿญ๐Ÿฏ: Mini Summit โ€” Mountain View ๐Ÿ”น ๐—ก๐—ผ๐˜ƒ๐—ฒ๐—บ๐—ฏ๐—ฒ๐—ฟ ๐Ÿฎ๐Ÿฐ: Meetup โ€” Paris ๐Ÿ”— RSVP and details: luma.com/openlakehouse A look back at our recent Open Lakehouse AI Amsterdam meetupโ€”huge thanks to the speakers, volunteers, and everyone who joined us! ๐Ÿ‘‡๐Ÿ“ธ #openlakehouse #ai #lakehouse #unitycatalog #apacheiceberg #deltalake #apachespark #oss #openlakehouseai
1
1
638
#UnityCatalog helps avoid data silos by offering a single unified system to manage, govern, and organize all your data and AI assetsโ€”without locking you into a specific table format or query engine. ๐Ÿš€ Define access rules once, keep metadata clean and centralized, and easily trace how data flows through your organization. Teams can continue to use their preferred tools and table formats, without compromising on efficiency or security. ๐Ÿ”— Learn more: unitycatalog.io/blogs/avoid-โ€ฆ #opensource #oss #unitycatalog #silos
96
Unity Catalog retweeted
What happens when robust, deterministic lakehouse design meets the power of AI agents? ๐Ÿ‘‰ Reduced tool fragmentation ๐Ÿ‘‰ Safer, reproducible automation ๐Ÿ‘‰ Data teams enabled for true innovation On Tuesday, October 7 at 9AM PT,ย @Bauplan_labsย foundersย @GreCo_CiRo and @jacopotagliabue willย break down how function-based execution and Git-for-Data semantics pave the way for autonomous agentic workflowsโ€”cutting complexity and multiplying impact. ๐Ÿ”— Register here to reserve your spot:ย luma.com/OLAI-107 #openlakehouse #opensource #oss #aiagents
2
4
430
Unity Catalog retweeted
At Open Lakehouse AI Amsterdam last month, @Sammy_Sidhu โ€”co-creator of the @daftengine multi-modal query engine and CEO of Eventualโ€”shared how his open source journey began.ย ๐Ÿš€ Ready to shape the future of data, open source, and AI together? Join us at our next Open Lakehouse AI events! ๐Ÿ”น Oct 7: Webinar โ€” From Functions to AI Agents: Reimagining the Lakehouse for an Agentic Future ๐Ÿ”น Nov 13: Mini Summit | Mountain View, CA ๐Ÿ”น Nov 24: Open Lakehouse AI | Paris, France Secure your spot: register now on Luma! ๐Ÿ”— luma.com/open-lakehouse #OpenLakehouseAI #OpenSource #Daft #DeltaLake #Lakehouse #OSS
3
4
443
Unity Catalog retweeted
๐Ÿš€ Open Lakehouse AI events are dedicated to advancing open lakehouse and AI through adoption, collaboration, and sharing real-world use cases. Explore whatโ€™s coming up to learn, connect, and collaborate with a global community passionate about shaping the future of data and AI. ๐Ÿ”น ๐—ข๐—ฐ๐˜๐—ผ๐—ฏ๐—ฒ๐—ฟ ๐Ÿณ: Webinar | From Functions to AI Agents: Reimagining the Lakehouse for an Agentic Future ๐Ÿ”น ๐—ก๐—ผ๐˜ƒ๐—ฒ๐—บ๐—ฏ๐—ฒ๐—ฟ ๐Ÿญ๐Ÿฏ: Mini Summit | Mountain View ๐Ÿ”น ๐—ก๐—ผ๐˜ƒ๐—ฒ๐—บ๐—ฏ๐—ฒ๐—ฟ ๐Ÿฎ๐Ÿฐ: Meetup | Paris ๐Ÿ”— RSVP & learn more at luma.com/open-lakehouse #opensource #lakehouse #openlakehouse #ai #deltalake #oss
4
13
2,400