Databricks open-sources Unity Catalog

2024-06-13 · Publisher : Databricks · Domain : Cloud & DevOps

Databricks open-sources Unity Catalog (Apache 2.0 licence, hosted by LF AI & Data): a data and AI governance catalog compatible with the Hive and Iceberg REST APIs.

### Background On 13 June 2024, at its Data + AI Summit, **Databricks** open-sourced **Unity Catalog**, which it presents as "the industry's first open source catalog for data and AI governance across clouds, data formats, and data platforms" (vendor wording). The code is under the **Apache 2.0** licence and hosted by the LF AI & Data foundation, part of the Linux Foundation. ### Key facts - Built on an OpenAPI specification, compatible with the Apache Hive metastore and the Apache Iceberg REST catalog API. - Supports Delta Lake, Iceberg (via UniForm), Parquet, CSV and JSON formats. - Version 0.1 manages tables, volumes (unstructured data) and AI functions or tools, with credential vending for centralized governance. - Partners named: AWS, Microsoft Azure, Google Cloud, NVIDIA, Salesforce, DuckDB, LangChain, dbt Labs, Fivetran, Confluent, Informatica, among others. - Databricks customers need to change nothing; external clients access tables through open REST APIs, with no external access fees. ### Why it matters Governance (who sees what, which data feeds which model) becomes the layer where lock-in is played out, after the table format. By opening its catalog, Databricks answers Snowflake, which had announced Polaris ten days earlier. For IT leaders the issue is portability: an open catalog and standard APIs make it possible to change compute engine without redoing governance. ### Takeaway After the table format, the catalog becomes a ground for openness, under competitive pressure between Databricks and Snowflake.

Tags : Données, Open Source, Licence, États-Unis