Unity Catalog has gained significant attention in the Databricks community recently, specifically around August 2022. Since its launch, it has emerged as the favored option among Databricks users due to its seamless integration with other components of the ecosystem.
What exactly is the Databricks Unity Catalog?
Unity Catalog is a unified governance solution for data and AI assets on Databricks Lakehouse. Unity Catalog provides centralized access control, auditing, lineage, and data discovery capabilities across Databricks workspaces.
The Unity Catalog Object Model
In Unity Catalog, the hierarchy of primary data objects flows from metastore to table or volume:
Metastore is the top-level container. Your data is organized into a three-level namespace (catalog.schema.table) that is exposed by each metastore. The catalog, which is the top tier of the object hierarchy, is where your data assets are arranged. Schemas, also referred to as databases, contain tables and views, and are the second tier of the object hierarchy.
The categories (tables, views, and volumes) are at the bottom of the data object hierarchy. Governance for non-tabular data is provided by volumes. As models, registered models can also be maintained in Unity Catalog and are at the bottom of the object hierarchy, although they are not technically data assets.
Benefits of Unity Catalog
The key benefits of Databricks Unity Catalog can be grouped into four main reasons: data discovery, governance, lineage, and sharing. This section will explore the capabilities of Unity Catalog within these four areas.
- Data Discovery
Unity Catalog offers the benefits of structured metadata organization alongside a robust search interface. While it provides access to search metadata, it also ensures that access is restricted based on the privileges and permissions of the logged-in user, thereby ensuring metadata-level security. Below is an illustration of the search interface:
Unity Catalog creates a “unified and secure search experience” with its data discovery features.
- Data Governance
When an organization utilizes a data platform like Databricks, there arises a necessity for data isolation boundaries between various environments (e.g., development and production) or among different organizational units.
Isolation standards may vary depending on your organization, but typically encompass the following expectations:
- Users can access data only according to specified access rules.
- Data management is limited to designated individuals or teams.
- Data is physically segregated in storage.
- Data access is restricted to specified environments.
The requirement for data isolation often results in siloed environments, which can impede both data governance and collaboration. Databricks addresses this challenge through Unity Catalog, offering a range of data isolation options while upholding a unified data governance platform.
For identity and access management, Databricks offers various user types, including service principals, users, and groups. These entities can establish a trust relationship with Databricks workspaces, resulting in identity federation.
With Unity Catalog, access controls based on rows and columns can be managed using pure SQL. Further granularity in access control can be achieved with attribute-based access control.
- Data Lineage
Data lineage is becoming increasingly important for several data engineering use cases, such as tracking and monitoring jobs, debugging failures, understanding complex workflows, tracing transformation rules, etc. Unity Catalog provides lineage not only at a table level, but also at a row level, allowing you to track which application is using which data — ideal for PII/GDPR data analysis and governance.
Lineage data contains vital insights into your company’s data flow. Unity Catalog adopts a similar approach to safeguard this information against unauthorized access. It does this by utilizing a governance model that limits access to data lineage based on the privileges of logged-in users. Undoubtedly, ensuring the security of your data holds significant importance.
- Delta Sharing
Delta Sharing serves as an open protocol facilitating secure data exchange with external organizations, irrespective of their chosen computing platforms. It enables real-time sharing of table collections stored within a Unity Catalog metastore without necessitating data duplication, allowing data recipients to promptly engage with the most current data version.
It is a highly transparent way of sharing data that not only reduces the workload of your data team but also helps them monitor and control access to data with clarity.
There are three components to Delta Sharing:
- Providers
A provider is an entity which has made data available for sharing.
- Shares
A share defines a logical grouping for the tables you intend to share.
- Recipients
A recipient identifies an organization with which you want to share any number of shares.
You do not need Unity Catalog to share (as a provider) or consume shared data (as a recipient). However, Unity Catalog provides benefits such as support for non-tabular and AI asset sharing, out-of-the-box governance, simplicity, and query performance.
Along with that, Data providers on Unity Catalog-enabled Databricks workspaces can use Databricks audit logging and system tables to monitor the creation and modification of shares and recipients, and can monitor recipient activity on shares.
In conclusion to its robust features and capabilities, Databricks Unity Catalog stands out for its user-friendly interface and seamless integration with existing Databricks workflows. Whether you’re a data engineer, data scientist, or business analyst, Unity Catalog streamlines data management tasks and enhances collaboration within your organization. Its intuitive design and extensive documentation make it easy to onboard new users and leverage its full potential. With Unity Catalog, Databricks users can confidently navigate the complexities of data governance, lineage tracking, and sharing, empowering them to make informed decisions and derive actionable insights from their data.
