Set up Databricks | Atlan Documentation

Atlan supports extracting the following for your Databricks assets using system tables : usage and popularity metrics Enable system.access schema You need your account admin to enable the system.access schema using the SystemSchemas API. This enables Atlan to extract lineage using system tables.


Set up Databricks | Atlan Documentation

In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore in your Databricks environment for which you want to extract lineage.


Set up Databricks | Atlan Documentation

To verify that system schemas are enabled for each schema, follow the steps in Databricks documentation : List system schemas using the SystemSchemas API to check the status. If enabled for any given schema, the state is EnableCompleted.


Set up Databricks | Atlan Documentation

This confirms that the schema has been enabled for that specific metastore. Atlan can only extract lineage using system tables when the state is marked as EnableCompleted.


Set up Databricks | Atlan Documentation

(Optional) enable system.information_schema.table To generate lineage with the target type set as PATH for a table, Atlan uses metadata from system.information_schema.table to resolve table paths and dependencies. To enable this, you must grant the following permissions on the relevant catalog, schema, and tables.


Set up Databricks | Atlan Documentation

You must be a metastore admin, have the MANAGE privilege on the object, or be the owner of the catalog, schema, or table to grant these permissions. In Atlan, one Databricks connection corresponds to one metastore.


Set up Databricks | Atlan Documentation

Repeat the following process for each metastore from which you want to extract lineage. Open Catalog Explorer in your Databricks workspace.


Set up Databricks | Atlan Documentation

Navigate to the catalog (for example, main ) and then to the appropriate schema (for example, sales ). Click the Permissions tab.


Set up Databricks | Atlan Documentation

Click Grant. Enter the user or group name (principal).


Set up Databricks | Atlan Documentation

Assign the following permissions: USAGE on the catalog USAGE on the schema SELECT on each relevant table USAGE on the catalog USAGE on the schema SELECT on each relevant table Click Grant to apply the changes. These privileges enable Atlan to read table definitions and other metadata from the metastore.


Set up Databricks | Atlan Documentation

(Optional) enable system.query schema This is only required if you also want to extract usage and popularity metrics from Databricks. You need your account admin to enable the system.query schema using the SystemSchemas API.


Set up Databricks | Atlan Documentation

This enables Atlan to mine query history using system tables for usage and popularity metrics. To verify that system schemas is enabled for each schema, follow the steps in Databricks documentation.


Set up Databricks | Atlan Documentation

If enabled for any given schema, the state is EnableCompleted. ð ª Did you know?


Set up Databricks | Atlan Documentation

Can't grant SELECT permissions on the system tables in system.access and system.query? Skip the previous steps and create cloned views in a separate catalog and schema.


Set up Databricks | Atlan Documentation

See Create cloned views of system tables. Grant permissions Atlan supports extracting Databricks lineage and usage and popularity metrics using system tables for all three authentication methods.


Set up Databricks | Atlan Documentation

Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following permissions: CAN_USE on a SQL warehouse USE_CATALOG on system catalog USE SCHEMA on system.access schema USE SCHEMA on system.query schema (tomine query history for usage and popularity metrics) SELECT on the following tables: system.query.history (to mine query history for usage and popularity metrics) system.access.table_lineage system.access.column_lineage system.query.history (to mine query history for usage and popularity metrics) You need to create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection.


Set up Databricks | Atlan Documentation

ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query?


Set up Databricks | Atlan Documentation

Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables.


Set up Databricks | Atlan Documentation

(Optional) Create cloned views of system tables When you don't want to grant access to system tables directly, you can create cloned views to expose lineage and popularity metrics through a separate schema. Follow these steps to set up cloned views: Create a catalog and schema to store cloned views.


Set up Databricks | Atlan Documentation

Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema. Create a catalog and schema to store cloned views.


Set up Databricks | Atlan Documentation

Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. column_lineage AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

access. column_lineage ; CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. table_lineage AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

access. table_lineage ; Replace and with the catalog and schema names used in your environment.


Set up Databricks | Atlan Documentation

Popularity metrics CREATE OR REPLACE VIEW. .


Set up Databricks | Atlan Documentation

query_history AS SELECT * FROM system. query.


Set up Databricks | Atlan Documentation

history ; Replace and with the catalog and schema names used in your environment. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. query_history AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

query. history ; Replace and with the catalog and schema names used in your environment.


Set up Databricks | Atlan Documentation

Replace and with the catalog and schema names used in your environment. Grant the following permissions to enable access to the cloned views: CAN_USE on a SQL warehouse USE CATALOG on the catalog (for example, ) USE SCHEMA and SELECT on the schema (for example,.


Set up Databricks | Atlan Documentation

) You must create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection.


Set up Databricks | Atlan Documentation

Locate warehouse ID To extract lineage and usage and popularity metrics using system tables, you will also need the warehouse ID of your SQL warehouse. To locate the warehouse ID: Log in to your Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the left menu of your workspace, click SQL Warehouses. On the Compute page, select the warehouse you want to use.


Set up Databricks | Atlan Documentation

(Optional) Grant view permissions to access Databricks entities via APIs Atlan uses Databricks REST APIs to extract metadata for Notebooks, Queries, Jobs, and Pipelines. This information helps to understand which Databricks enitity was used to create a lineage between assets.


Set up Databricks | Atlan Documentation

Use the steps below for each object type to grant CAN VIEW permission to the Databricks user or service principal configured in your integration: Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders.


Set up Databricks | Atlan Documentation

Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders.


Set up Databricks | Atlan Documentation

Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries.


Set up Databricks | Atlan Documentation

Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object.


Set up Databricks | Atlan Documentation

For more information, see Control Access to a Job. Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly.


Set up Databricks | Atlan Documentation

Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job.


Set up Databricks | Atlan Documentation

Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions.


Set up Databricks | Atlan Documentation

(Optional) Grant permissions for views and materialized views Atlan requires the following permissions to to extract view definitions from and generate lineagefor views and materialized views: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the Catalog Explorer , select the catalog you want to extract view definitions from and generate lineage for in Atlan. From the tabs at the top, click the Permissions tab, and then click Grant.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Select the user or service principal under Principals. Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Select the user or service principal under Principals.


Set up Databricks | Atlan Documentation

Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Click Grant to apply the permissions. Repeat steps 3 6 for each catalog you want to crawl in Atlan.


Set up Databricks | Atlan Documentation

SELECT permission is required to extract the definitions of views and materialized views. If you prefer not to grant SELECT at the catalog level, you can grant it on individual views and materialized views instead.


Set up Databricks | Atlan Documentation

(Optional) Grant permissions to mine query history To mine query history using REST API, you will need to assign the CAN MANAGE permission on your SQL warehouses to the user or service principal. To grant permissions to mine query history: Log in to your Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the left menu of your workspace, click SQL Warehouses. On the Compute page, for each SQL warehouse you want to mine query history, click the 3-dot icon and then click Permissions.


Set up Databricks | Atlan Documentation

Expand the Can use permissions dropdown and then select Can manage. This permission enables the service principal to view all queries for the warehouse.


Set up Databricks | Atlan Documentation

Click Add to assign the CAN MANAGE permission to the service principal. Personal access token authentication AWS service principal authentication Azure service principal authentication Grant permissions to crawl metadata (Optional) Grant permissions to query and preview data (Optional) Grant permissions to import and update tags (Optional) Grant permissions to extract lineage and usage from system tables (Optional) Grant view permissions to access Databricks entities via APIs (Optional) Grant permissions for views and materialized views (Optional) Grant permissions to mine query history


Set up Databricks | Atlan Documentation

Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods: Personal access token authentication AWS service principal authentication Azure service principal authentication Personal access token authentication Check that you have Admin and Databricks SQL access for the Databricks workspace.


Set up Databricks | Atlan Documentation

This is required for both cluster options described below. If you don't have this access, contact your Databricks administrator.


Set up Databricks | Atlan Documentation

Grant user access to workspace To grant workspace access to the user creating a personal access token: From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the user. From the tabs along the top of your workspace page, click the Permissions tab.


Set up Databricks | Atlan Documentation

In the upper right of the Permissions page, click Add permissions. In the Add permissions dialog, enter the following details: For User, group, or service principal , select the user to grant access.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the user to grant access.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. Generate a personal access token You can generate a personal access token in your Databricks workspace to the authenticate the integration in Atlan.


Set up Databricks | Atlan Documentation

To generate a personal access token: From the top right of your Databricks workspace, click your Databricks username, and then from the dropdown, click User Settings. Under the Settings menu, click Developer.


Set up Databricks | Atlan Documentation

On the Developer page, next to Access tokens , click Manage. On the Access tokens page, click the Generate new token button.


Set up Databricks | Atlan Documentation

In the Generate new token dialog: For Comment , enter a description of the token's intended use - for example, Atlan crawler. For Lifetime (days) , consider removing the number.


Set up Databricks | Atlan Documentation

This enables the token to be used indefinitely - it won't need to be refreshed. Important!


Set up Databricks | Atlan Documentation

If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate.


Set up Databricks | Atlan Documentation

For Comment , enter a description of the token's intended use - for example, Atlan crawler. For Comment , enter a description of the token's intended use - for example, Atlan crawler.


Set up Databricks | Atlan Documentation

For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed.


Set up Databricks | Atlan Documentation

Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

At the bottom of the dialog, click Generate. Copy and save the generated token in a secure location, and then click Done.


Set up Databricks | Atlan Documentation

Select a cluster Atlan recommends using serverless SQL warehouses for instant compute availability. To enable serverless SQL warehouses, refer to Databricks documentation for AWS Databricks workspaces or Microsoft documentation for Azure Databricks workspaces.


Set up Databricks | Atlan Documentation

You can set up personal access token authentication for your Databricks instance using one of the following cluster options: SQL warehouse (formerly SQL endpoint) To confirm an all-purpose interactive cluster is configured: From the left menu of any page of your Databricks instance, click Compute. Under the All-purpose clusters tab, verify you have a cluster defined.


Set up Databricks | Atlan Documentation

Click the link under the Name column of the table to open your cluster. Under the Configuration tab, verify the Autopilot options to Terminate after.


Set up Databricks | Atlan Documentation

minutes is enabled. At the bottom of the Configuration tab, expand the Advanced options expandable.


Set up Databricks | Atlan Documentation

Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path.


Set up Databricks | Atlan Documentation

To confirm a SQL warehouse is configured: From the left menu of any page of your Databricks instance, open the dropdown just below the databricks logo and change to SQL. From the refreshed left menu, click SQL Warehouses.


Set up Databricks | Atlan Documentation

Click the link under the Name column of the table to open your SQL warehouse. Under the Connection details tab, confirm that all of the fields are populated and copy them for use in crawling: Server hostname , Port , and HTTP path.


Set up Databricks | Atlan Documentation

AWS service principal authentication You need your AWS Databricks account admin to create a service principal and manage OAuth credentials for the service principal and your AWS Databricks workspace admin to add the service principal to your AWS Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Create a service principal You can create a service principal directly in your Databricks account or from a Databricks workspace.


Set up Databricks | Atlan Documentation

Identity federation enabled on your workspaces: Databricks recommends creating the service principal in the account and assigning it to workspaces. Identity federation disabled on your workspaces: Databricks recommends that you create your service principal from a workspace.


Set up Databricks | Atlan Documentation

To create a service principal from your Databricks account, with identify federation enabled: Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management.


Set up Databricks | Atlan Documentation

From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal.


Set up Databricks | Atlan Documentation

On the Add service principal page, enter a name for the service principal and then click Add. Once the service principal has been created, you can assign it to your identity federated workspace.


Set up Databricks | Atlan Documentation

From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab.


Set up Databricks | Atlan Documentation

In the upper right of the Permissions page, click Add permissions. In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the service principal you created.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. To create a service principal from a Databricks workspace, with identity federation disabled: Log in to your AWS Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings. In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access.


Set up Databricks | Atlan Documentation

On the Identity and access page, under Management and permissions , next to Service principals , click Manage. In the upper right of the Service principals page, click Add service principal.


Set up Databricks | Atlan Documentation

In the Add service principal dialog, click the Add new button. For New service principal display name , enter a name for the service principal and then click Add.


Set up Databricks | Atlan Documentation

Create an OAuth secret for the service principal You need to create an OAuth secret to authenticate to Databricks REST APIs. To create an OAuth secret for the service principal : Log in to your Databricks account console as an account admin.


Set up Databricks | Atlan Documentation

Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management.


Set up Databricks | Atlan Documentation

From the left menu of the account console, click User management. From the tabs along the top of the User management page, click the Service principals tab.


Set up Databricks | Atlan Documentation

From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, select the service principal you created.


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, select the service principal you created. On the service principal page, under OAuth secrets , click Generate secret.


Set up Databricks | Atlan Documentation

On the service principal page, under OAuth secrets , click Generate secret. From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location.


Set up Databricks | Atlan Documentation

danger Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal.


Set up Databricks | Atlan Documentation

From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. Note that this secret is only revealed once during creation.


Set up Databricks | Atlan Documentation

The client ID is the same as the application ID of the service principal. Once you've copied the client ID and secret, click Done.


Set up Databricks | Atlan Documentation

Once you've copied the client ID and secret, click Done. Azure service principal authentication You need your Azure Databricks account admin to create a service principal and your Azure Databricks workspace admin to add the service principal to your Azure Databricks workspace - you may not have access yourself.


Set up Databricks | Atlan Documentation

You need the following to authenticate the connection in Atlan: Client ID (application ID) Tenant ID (directory ID) Create a service principal To use service principals on Azure Databricks , an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application and then add it to the Azure Databricks workspace to use as a service principal. To create a service principal: Sign in to the Azure portal.


Set up Databricks | Atlan Documentation

If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to create the service principal. Click**+ Add and select App registration**.


Set up Databricks | Atlan Documentation

For_Name_, enter a name for the application. In the_Supported account types_section, select Accounts in this organizational directory only (Single tenant) and then click Register.


Set up Databricks | Atlan Documentation

Application (client) ID Directory (tenant) ID To generate a client secret, within_Manage_, click Certificates & secrets. On the_Client secrets_tab, click New client secret.


Set up Databricks | Atlan Documentation

In the_Add a client secret_dialog, enter the following details: For Description , enter a description for the client secret. For_Expires_, select an expiry time period for the client secret and then click Add.


Set up Databricks | Atlan Documentation

Copy and store the client secret's_Value_in a secure place. Add a service principal to your account To add a service principal to your Azure Databricks account: Log in to your Azure Databricks account console as an account admin.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, click Add service principal. On the Add service principal page, enter a name for the service principal.


Set up Databricks | Atlan Documentation

Under UUID , paste the Application (client) ID for the service principal. Click Add.


Set up Databricks | Atlan Documentation

Assign a service principal to a workspace To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page.


Set up Databricks | Atlan Documentation

To assign a service principal to your Azure Databricks account: Log in to your Databricks account console as an account admin. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal.


Set up Databricks | Atlan Documentation

From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions.


Set up Databricks | Atlan Documentation

In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User.


Set up Databricks | Atlan Documentation

For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User.


Set up Databricks | Atlan Documentation

To assign a service principal to your Azure Databricks workspace: Log in to your Azure Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings.


Set up Databricks | Atlan Documentation

In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access. On the Identity and access page, under Management and permissions , next to Service principals , click Manage.


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, click Add service principal. In the Add service principal dialog, click the Add new button.


Set up Databricks | Atlan Documentation

For New service principal display name , paste the Application (client) ID for the service principal , enter a display name, and then click Add. Grant permissions to crawl metadata You must have a Unity Catalog-enabled Databricks workspace to crawl metadata in Atlan.


Set up Databricks | Atlan Documentation

To extract metadata, you can grant the BROWSE privilege , currently in public preview. You no longer require the Data Reader preset that granted the following privileges on objects in the catalog - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT.


Set up Databricks | Atlan Documentation

To grant permissions to a user or service principal: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the left menu of the Catalog Explorer page, select the catalog you want to crawl in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege.


Set up Databricks | Atlan Documentation

At the bottom of the dialog, click Grant. Under Principals , click the dropdown and then select the user or service principal.


Set up Databricks | Atlan Documentation

Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant.


Set up Databricks | Atlan Documentation

(Optional) Repeat steps 3-5 for each catalog you want to crawl in Atlan. System tables extraction method To crawl metadata via system tables, you must have a Unity Catalog-enabled workspace and a configured SQL warehouse.


Set up Databricks | Atlan Documentation

Follow these steps to extract metadata using system tables: Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Cross-workspace extraction To crawl metadata from all workspaces within a Databricks metastore using a single connection, see Set up cross-workspace extraction for instructions. (Optional) Grant permissions to query and preview data Atlan currently only supports querying data and viewing sample data preview for the personal access token authentication method.


Set up Databricks | Atlan Documentation

To grant permissions to query data and preview example data: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the left menu of the Catalog Explorer page, select the catalog you want to query and preview data from in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog.


Set up Databricks | Atlan Documentation

Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT. At the bottom of the dialog, click Grant.


Set up Databricks | Atlan Documentation

Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

(Optional) Repeat steps 3-5 for each catalog you want to query and preview data from in Atlan. (Optional) Grant permissions to import and update tags To import Databricks tags , you must have a Unity Catalog-enabled workspace and a SQL warehouse configured.


Set up Databricks | Atlan Documentation

Atlan supports importing Databricks tags using system tables for all three authentication methods. Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following privileges: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalog_tags system.information_schema.schema_tags system.information_schema.table_tags system.information_schema.column_tags To push tags updated for assets in Atlan to Databricks, you need to grant the following privileges : APPLY TAG on the object USE CATALOG on the object's parent catalog USE SCHEMA on the object's parent schema (Optional) Grant permissions to extract lineage and usage from system tables You must have a Unity Catalog-enabled workspace to use system tables.


Set up Databricks | Atlan Documentation

Atlan supports extracting the following for your Databricks assets using system tables : usage and popularity metrics Enable system.access schema You need your account admin to enable the system.access schema using the SystemSchemas API. This enables Atlan to extract lineage using system tables.


Set up Databricks | Atlan Documentation

In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore in your Databricks environment for which you want to extract lineage.


Set up Databricks | Atlan Documentation

To verify that system schemas are enabled for each schema, follow the steps in Databricks documentation : List system schemas using the SystemSchemas API to check the status. If enabled for any given schema, the state is EnableCompleted.


Set up Databricks | Atlan Documentation

This confirms that the schema has been enabled for that specific metastore. Atlan can only extract lineage using system tables when the state is marked as EnableCompleted.


Set up Databricks | Atlan Documentation

(Optional) enable system.information_schema.table To generate lineage with the target type set as PATH for a table, Atlan uses metadata from system.information_schema.table to resolve table paths and dependencies. To enable this, you must grant the following permissions on the relevant catalog, schema, and tables.


Set up Databricks | Atlan Documentation

You must be a metastore admin, have the MANAGE privilege on the object, or be the owner of the catalog, schema, or table to grant these permissions. In Atlan, one Databricks connection corresponds to one metastore.


Set up Databricks | Atlan Documentation

Repeat the following process for each metastore from which you want to extract lineage. Open Catalog Explorer in your Databricks workspace.


Set up Databricks | Atlan Documentation

Navigate to the catalog (for example, main ) and then to the appropriate schema (for example, sales ). Click the Permissions tab.


Set up Databricks | Atlan Documentation

Click Grant. Enter the user or group name (principal).


Set up Databricks | Atlan Documentation

Assign the following permissions: USAGE on the catalog USAGE on the schema SELECT on each relevant table USAGE on the catalog USAGE on the schema SELECT on each relevant table Click Grant to apply the changes. These privileges enable Atlan to read table definitions and other metadata from the metastore.


Set up Databricks | Atlan Documentation

(Optional) enable system.query schema This is only required if you also want to extract usage and popularity metrics from Databricks. You need your account admin to enable the system.query schema using the SystemSchemas API.


Set up Databricks | Atlan Documentation

This enables Atlan to mine query history using system tables for usage and popularity metrics. To verify that system schemas is enabled for each schema, follow the steps in Databricks documentation.


Set up Databricks | Atlan Documentation

If enabled for any given schema, the state is EnableCompleted. ð ª Did you know?


Set up Databricks | Atlan Documentation

Can't grant SELECT permissions on the system tables in system.access and system.query? Skip the previous steps and create cloned views in a separate catalog and schema.


Set up Databricks | Atlan Documentation

See Create cloned views of system tables. Grant permissions Atlan supports extracting Databricks lineage and usage and popularity metrics using system tables for all three authentication methods.


Set up Databricks | Atlan Documentation

Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following permissions: CAN_USE on a SQL warehouse USE_CATALOG on system catalog USE SCHEMA on system.access schema USE SCHEMA on system.query schema (tomine query history for usage and popularity metrics) SELECT on the following tables: system.query.history (to mine query history for usage and popularity metrics) system.access.table_lineage system.access.column_lineage system.query.history (to mine query history for usage and popularity metrics) You need to create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection.


Set up Databricks | Atlan Documentation

ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query?


Set up Databricks | Atlan Documentation

Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables.


Set up Databricks | Atlan Documentation

(Optional) Create cloned views of system tables When you don't want to grant access to system tables directly, you can create cloned views to expose lineage and popularity metrics through a separate schema. Follow these steps to set up cloned views: Create a catalog and schema to store cloned views.


Set up Databricks | Atlan Documentation

Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema. Create a catalog and schema to store cloned views.


Set up Databricks | Atlan Documentation

Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. column_lineage AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

access. column_lineage ; CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. table_lineage AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

access. table_lineage ; Replace and with the catalog and schema names used in your environment.


Set up Databricks | Atlan Documentation

Popularity metrics CREATE OR REPLACE VIEW. .


Set up Databricks | Atlan Documentation

query_history AS SELECT * FROM system. query.


Set up Databricks | Atlan Documentation

history ; Replace and with the catalog and schema names used in your environment. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. query_history AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

query. history ; Replace and with the catalog and schema names used in your environment.


Set up Databricks | Atlan Documentation

Replace and with the catalog and schema names used in your environment. Grant the following permissions to enable access to the cloned views: CAN_USE on a SQL warehouse USE CATALOG on the catalog (for example, ) USE SCHEMA and SELECT on the schema (for example,.


Set up Databricks | Atlan Documentation

) You must create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection.


Set up Databricks | Atlan Documentation

Locate warehouse ID To extract lineage and usage and popularity metrics using system tables, you will also need the warehouse ID of your SQL warehouse. To locate the warehouse ID: Log in to your Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the left menu of your workspace, click SQL Warehouses. On the Compute page, select the warehouse you want to use.


Set up Databricks | Atlan Documentation

(Optional) Grant view permissions to access Databricks entities via APIs Atlan uses Databricks REST APIs to extract metadata for Notebooks, Queries, Jobs, and Pipelines. This information helps to understand which Databricks enitity was used to create a lineage between assets.


Set up Databricks | Atlan Documentation

Use the steps below for each object type to grant CAN VIEW permission to the Databricks user or service principal configured in your integration: Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders.


Set up Databricks | Atlan Documentation

Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders.


Set up Databricks | Atlan Documentation

Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries.


Set up Databricks | Atlan Documentation

Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object.


Set up Databricks | Atlan Documentation

For more information, see Control Access to a Job. Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly.


Set up Databricks | Atlan Documentation

Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job.


Set up Databricks | Atlan Documentation

Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions.


Set up Databricks | Atlan Documentation

(Optional) Grant permissions for views and materialized views Atlan requires the following permissions to to extract view definitions from and generate lineagefor views and materialized views: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the Catalog Explorer , select the catalog you want to extract view definitions from and generate lineage for in Atlan. From the tabs at the top, click the Permissions tab, and then click Grant.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Select the user or service principal under Principals. Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Select the user or service principal under Principals.


Set up Databricks | Atlan Documentation

Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Click Grant to apply the permissions. Repeat steps 3 6 for each catalog you want to crawl in Atlan.


Set up Databricks | Atlan Documentation

SELECT permission is required to extract the definitions of views and materialized views. If you prefer not to grant SELECT at the catalog level, you can grant it on individual views and materialized views instead.


Set up Databricks | Atlan Documentation

(Optional) Grant permissions to mine query history To mine query history using REST API, you will need to assign the CAN MANAGE permission on your SQL warehouses to the user or service principal. To grant permissions to mine query history: Log in to your Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the left menu of your workspace, click SQL Warehouses. On the Compute page, for each SQL warehouse you want to mine query history, click the 3-dot icon and then click Permissions.


Set up Databricks | Atlan Documentation

Expand the Can use permissions dropdown and then select Can manage. This permission enables the service principal to view all queries for the warehouse.


Set up Databricks | Atlan Documentation

Click Add to assign the CAN MANAGE permission to the service principal. Personal access token authentication AWS service principal authentication Azure service principal authentication Grant permissions to crawl metadata (Optional) Grant permissions to query and preview data (Optional) Grant permissions to import and update tags (Optional) Grant permissions to extract lineage and usage from system tables (Optional) Grant view permissions to access Databricks entities via APIs (Optional) Grant permissions for views and materialized views (Optional) Grant permissions to mine query history


Set up Databricks | Atlan Documentation

Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods: Personal access token authentication AWS service principal authentication Azure service principal authentication Personal access token authentication Check that you have Admin and Databricks SQL access for the Databricks workspace.


Set up Databricks | Atlan Documentation

This is required for both cluster options described below. If you don't have this access, contact your Databricks administrator.


Set up Databricks | Atlan Documentation

Grant user access to workspace To grant workspace access to the user creating a personal access token: From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the user. From the tabs along the top of your workspace page, click the Permissions tab.


Set up Databricks | Atlan Documentation

In the upper right of the Permissions page, click Add permissions. In the Add permissions dialog, enter the following details: For User, group, or service principal , select the user to grant access.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the user to grant access.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. Generate a personal access token You can generate a personal access token in your Databricks workspace to the authenticate the integration in Atlan.


Set up Databricks | Atlan Documentation

To generate a personal access token: From the top right of your Databricks workspace, click your Databricks username, and then from the dropdown, click User Settings. Under the Settings menu, click Developer.


Set up Databricks | Atlan Documentation

On the Developer page, next to Access tokens , click Manage. On the Access tokens page, click the Generate new token button.


Set up Databricks | Atlan Documentation

In the Generate new token dialog: For Comment , enter a description of the token's intended use - for example, Atlan crawler. For Lifetime (days) , consider removing the number.


Set up Databricks | Atlan Documentation

This enables the token to be used indefinitely - it won't need to be refreshed. Important!


Set up Databricks | Atlan Documentation

If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate.


Set up Databricks | Atlan Documentation

For Comment , enter a description of the token's intended use - for example, Atlan crawler. For Comment , enter a description of the token's intended use - for example, Atlan crawler.


Set up Databricks | Atlan Documentation

For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed.


Set up Databricks | Atlan Documentation

Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

At the bottom of the dialog, click Generate. Copy and save the generated token in a secure location, and then click Done.


Set up Databricks | Atlan Documentation

Select a cluster Atlan recommends using serverless SQL warehouses for instant compute availability. To enable serverless SQL warehouses, refer to Databricks documentation for AWS Databricks workspaces or Microsoft documentation for Azure Databricks workspaces.


Set up Databricks | Atlan Documentation

You can set up personal access token authentication for your Databricks instance using one of the following cluster options: SQL warehouse (formerly SQL endpoint) To confirm an all-purpose interactive cluster is configured: From the left menu of any page of your Databricks instance, click Compute. Under the All-purpose clusters tab, verify you have a cluster defined.


Set up Databricks | Atlan Documentation

Click the link under the Name column of the table to open your cluster. Under the Configuration tab, verify the Autopilot options to Terminate after.


Set up Databricks | Atlan Documentation

minutes is enabled. At the bottom of the Configuration tab, expand the Advanced options expandable.


Set up Databricks | Atlan Documentation

Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path.


Set up Databricks | Atlan Documentation

To confirm a SQL warehouse is configured: From the left menu of any page of your Databricks instance, open the dropdown just below the databricks logo and change to SQL. From the refreshed left menu, click SQL Warehouses.


Set up Databricks | Atlan Documentation

Click the link under the Name column of the table to open your SQL warehouse. Under the Connection details tab, confirm that all of the fields are populated and copy them for use in crawling: Server hostname , Port , and HTTP path.


Set up Databricks | Atlan Documentation

AWS service principal authentication You need your AWS Databricks account admin to create a service principal and manage OAuth credentials for the service principal and your AWS Databricks workspace admin to add the service principal to your AWS Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Create a service principal You can create a service principal directly in your Databricks account or from a Databricks workspace.


Set up Databricks | Atlan Documentation

Identity federation enabled on your workspaces: Databricks recommends creating the service principal in the account and assigning it to workspaces. Identity federation disabled on your workspaces: Databricks recommends that you create your service principal from a workspace.


Set up Databricks | Atlan Documentation

To create a service principal from your Databricks account, with identify federation enabled: Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management.


Set up Databricks | Atlan Documentation

From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal.


Set up Databricks | Atlan Documentation

On the Add service principal page, enter a name for the service principal and then click Add. Once the service principal has been created, you can assign it to your identity federated workspace.


Set up Databricks | Atlan Documentation

From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab.


Set up Databricks | Atlan Documentation

In the upper right of the Permissions page, click Add permissions. In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the service principal you created.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. To create a service principal from a Databricks workspace, with identity federation disabled: Log in to your AWS Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings. In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access.


Set up Databricks | Atlan Documentation

On the Identity and access page, under Management and permissions , next to Service principals , click Manage. In the upper right of the Service principals page, click Add service principal.


Set up Databricks | Atlan Documentation

In the Add service principal dialog, click the Add new button. For New service principal display name , enter a name for the service principal and then click Add.


Set up Databricks | Atlan Documentation

Create an OAuth secret for the service principal You need to create an OAuth secret to authenticate to Databricks REST APIs. To create an OAuth secret for the service principal : Log in to your Databricks account console as an account admin.


Set up Databricks | Atlan Documentation

Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management.


Set up Databricks | Atlan Documentation

From the left menu of the account console, click User management. From the tabs along the top of the User management page, click the Service principals tab.


Set up Databricks | Atlan Documentation

From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, select the service principal you created.


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, select the service principal you created. On the service principal page, under OAuth secrets , click Generate secret.


Set up Databricks | Atlan Documentation

On the service principal page, under OAuth secrets , click Generate secret. From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location.


Set up Databricks | Atlan Documentation

danger Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal.


Set up Databricks | Atlan Documentation

From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. Note that this secret is only revealed once during creation.


Set up Databricks | Atlan Documentation

The client ID is the same as the application ID of the service principal. Once you've copied the client ID and secret, click Done.


Set up Databricks | Atlan Documentation

Once you've copied the client ID and secret, click Done. Azure service principal authentication You need your Azure Databricks account admin to create a service principal and your Azure Databricks workspace admin to add the service principal to your Azure Databricks workspace - you may not have access yourself.


Set up Databricks | Atlan Documentation

You need the following to authenticate the connection in Atlan: Client ID (application ID) Tenant ID (directory ID) Create a service principal To use service principals on Azure Databricks , an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application and then add it to the Azure Databricks workspace to use as a service principal. To create a service principal: Sign in to the Azure portal.


Set up Databricks | Atlan Documentation

If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to create the service principal. Click**+ Add and select App registration**.


Set up Databricks | Atlan Documentation

For_Name_, enter a name for the application. In the_Supported account types_section, select Accounts in this organizational directory only (Single tenant) and then click Register.


Set up Databricks | Atlan Documentation

Application (client) ID Directory (tenant) ID To generate a client secret, within_Manage_, click Certificates & secrets. On the_Client secrets_tab, click New client secret.


Set up Databricks | Atlan Documentation

In the_Add a client secret_dialog, enter the following details: For Description , enter a description for the client secret. For_Expires_, select an expiry time period for the client secret and then click Add.


Set up Databricks | Atlan Documentation

Copy and store the client secret's_Value_in a secure place. Add a service principal to your account To add a service principal to your Azure Databricks account: Log in to your Azure Databricks account console as an account admin.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, click Add service principal. On the Add service principal page, enter a name for the service principal.


Set up Databricks | Atlan Documentation

Under UUID , paste the Application (client) ID for the service principal. Click Add.


Set up Databricks | Atlan Documentation

Assign a service principal to a workspace To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page.


Set up Databricks | Atlan Documentation

To assign a service principal to your Azure Databricks account: Log in to your Databricks account console as an account admin. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal.


Set up Databricks | Atlan Documentation

From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions.


Set up Databricks | Atlan Documentation

In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User.


Set up Databricks | Atlan Documentation

For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User.


Set up Databricks | Atlan Documentation

To assign a service principal to your Azure Databricks workspace: Log in to your Azure Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings.


Set up Databricks | Atlan Documentation

In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access. On the Identity and access page, under Management and permissions , next to Service principals , click Manage.


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, click Add service principal. In the Add service principal dialog, click the Add new button.


Set up Databricks | Atlan Documentation

For New service principal display name , paste the Application (client) ID for the service principal , enter a display name, and then click Add. Grant permissions to crawl metadata You must have a Unity Catalog-enabled Databricks workspace to crawl metadata in Atlan.


Set up Databricks | Atlan Documentation

To extract metadata, you can grant the BROWSE privilege , currently in public preview. You no longer require the Data Reader preset that granted the following privileges on objects in the catalog - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT.


Set up Databricks | Atlan Documentation

To grant permissions to a user or service principal: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the left menu of the Catalog Explorer page, select the catalog you want to crawl in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege.


Set up Databricks | Atlan Documentation

At the bottom of the dialog, click Grant. Under Principals , click the dropdown and then select the user or service principal.


Set up Databricks | Atlan Documentation

Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant.


Set up Databricks | Atlan Documentation

(Optional) Repeat steps 3-5 for each catalog you want to crawl in Atlan. System tables extraction method To crawl metadata via system tables, you must have a Unity Catalog-enabled workspace and a configured SQL warehouse.


Set up Databricks | Atlan Documentation

Follow these steps to extract metadata using system tables: Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Cross-workspace extraction To crawl metadata from all workspaces within a Databricks metastore using a single connection, see Set up cross-workspace extraction for instructions. (Optional) Grant permissions to query and preview data Atlan currently only supports querying data and viewing sample data preview for the personal access token authentication method.


Set up Databricks | Atlan Documentation

To grant permissions to query data and preview example data: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the left menu of the Catalog Explorer page, select the catalog you want to query and preview data from in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog.


Set up Databricks | Atlan Documentation

Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT. At the bottom of the dialog, click Grant.


Set up Databricks | Atlan Documentation

Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

(Optional) Repeat steps 3-5 for each catalog you want to query and preview data from in Atlan. (Optional) Grant permissions to import and update tags To import Databricks tags , you must have a Unity Catalog-enabled workspace and a SQL warehouse configured.


Set up Databricks | Atlan Documentation

Atlan supports importing Databricks tags using system tables for all three authentication methods. Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following privileges: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalog_tags system.information_schema.schema_tags system.information_schema.table_tags system.information_schema.column_tags To push tags updated for assets in Atlan to Databricks, you need to grant the following privileges : APPLY TAG on the object USE CATALOG on the object's parent catalog USE SCHEMA on the object's parent schema (Optional) Grant permissions to extract lineage and usage from system tables You must have a Unity Catalog-enabled workspace to use system tables.


Set up Databricks | Atlan Documentation

Atlan supports extracting the following for your Databricks assets using system tables : usage and popularity metrics Enable system.access schema You need your account admin to enable the system.access schema using the SystemSchemas API. This enables Atlan to extract lineage using system tables.


Set up Databricks | Atlan Documentation

In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore in your Databricks environment for which you want to extract lineage.


Set up Databricks | Atlan Documentation

To verify that system schemas are enabled for each schema, follow the steps in Databricks documentation : List system schemas using the SystemSchemas API to check the status. If enabled for any given schema, the state is EnableCompleted.


Set up Databricks | Atlan Documentation

This confirms that the schema has been enabled for that specific metastore. Atlan can only extract lineage using system tables when the state is marked as EnableCompleted.


Set up Databricks | Atlan Documentation

(Optional) enable system.information_schema.table To generate lineage with the target type set as PATH for a table, Atlan uses metadata from system.information_schema.table to resolve table paths and dependencies. To enable this, you must grant the following permissions on the relevant catalog, schema, and tables.


Set up Databricks | Atlan Documentation

You must be a metastore admin, have the MANAGE privilege on the object, or be the owner of the catalog, schema, or table to grant these permissions. In Atlan, one Databricks connection corresponds to one metastore.


Set up Databricks | Atlan Documentation

Repeat the following process for each metastore from which you want to extract lineage. Open Catalog Explorer in your Databricks workspace.


Set up Databricks | Atlan Documentation

Navigate to the catalog (for example, main ) and then to the appropriate schema (for example, sales ). Click the Permissions tab.


Set up Databricks | Atlan Documentation

Click Grant. Enter the user or group name (principal).


Set up Databricks | Atlan Documentation

Assign the following permissions: USAGE on the catalog USAGE on the schema SELECT on each relevant table USAGE on the catalog USAGE on the schema SELECT on each relevant table Click Grant to apply the changes. These privileges enable Atlan to read table definitions and other metadata from the metastore.


Set up Databricks | Atlan Documentation

(Optional) enable system.query schema This is only required if you also want to extract usage and popularity metrics from Databricks. You need your account admin to enable the system.query schema using the SystemSchemas API.


Set up Databricks | Atlan Documentation

This enables Atlan to mine query history using system tables for usage and popularity metrics. To verify that system schemas is enabled for each schema, follow the steps in Databricks documentation.


Set up Databricks | Atlan Documentation

If enabled for any given schema, the state is EnableCompleted. ð ª Did you know?


Set up Databricks | Atlan Documentation

Can't grant SELECT permissions on the system tables in system.access and system.query? Skip the previous steps and create cloned views in a separate catalog and schema.


Set up Databricks | Atlan Documentation

See Create cloned views of system tables. Grant permissions Atlan supports extracting Databricks lineage and usage and popularity metrics using system tables for all three authentication methods.


Set up Databricks | Atlan Documentation

Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following permissions: CAN_USE on a SQL warehouse USE_CATALOG on system catalog USE SCHEMA on system.access schema USE SCHEMA on system.query schema (tomine query history for usage and popularity metrics) SELECT on the following tables: system.query.history (to mine query history for usage and popularity metrics) system.access.table_lineage system.access.column_lineage system.query.history (to mine query history for usage and popularity metrics) You need to create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection.


Set up Databricks | Atlan Documentation

ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query?


Set up Databricks | Atlan Documentation

Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables.


Set up Databricks | Atlan Documentation

(Optional) Create cloned views of system tables When you don't want to grant access to system tables directly, you can create cloned views to expose lineage and popularity metrics through a separate schema. Follow these steps to set up cloned views: Create a catalog and schema to store cloned views.


Set up Databricks | Atlan Documentation

Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema. Create a catalog and schema to store cloned views.


Set up Databricks | Atlan Documentation

Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. column_lineage AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

access. column_lineage ; CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. table_lineage AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

access. table_lineage ; Replace and with the catalog and schema names used in your environment.


Set up Databricks | Atlan Documentation

Popularity metrics CREATE OR REPLACE VIEW. .


Set up Databricks | Atlan Documentation

query_history AS SELECT * FROM system. query.


Set up Databricks | Atlan Documentation

history ; Replace and with the catalog and schema names used in your environment. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW.


Set up Databricks | Atlan Documentation

. query_history AS SELECT * FROM system.


Set up Databricks | Atlan Documentation

query. history ; Replace and with the catalog and schema names used in your environment.


Set up Databricks | Atlan Documentation

Replace and with the catalog and schema names used in your environment. Grant the following permissions to enable access to the cloned views: CAN_USE on a SQL warehouse USE CATALOG on the catalog (for example, ) USE SCHEMA and SELECT on the schema (for example,.


Set up Databricks | Atlan Documentation

) You must create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection.


Set up Databricks | Atlan Documentation

Locate warehouse ID To extract lineage and usage and popularity metrics using system tables, you will also need the warehouse ID of your SQL warehouse. To locate the warehouse ID: Log in to your Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the left menu of your workspace, click SQL Warehouses. On the Compute page, select the warehouse you want to use.


Set up Databricks | Atlan Documentation

(Optional) Grant view permissions to access Databricks entities via APIs Atlan uses Databricks REST APIs to extract metadata for Notebooks, Queries, Jobs, and Pipelines. This information helps to understand which Databricks enitity was used to create a lineage between assets.


Set up Databricks | Atlan Documentation

Use the steps below for each object type to grant CAN VIEW permission to the Databricks user or service principal configured in your integration: Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders.


Set up Databricks | Atlan Documentation

Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders.


Set up Databricks | Atlan Documentation

Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries.


Set up Databricks | Atlan Documentation

Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object.


Set up Databricks | Atlan Documentation

For more information, see Control Access to a Job. Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly.


Set up Databricks | Atlan Documentation

Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job.


Set up Databricks | Atlan Documentation

Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions.


Set up Databricks | Atlan Documentation

(Optional) Grant permissions for views and materialized views Atlan requires the following permissions to to extract view definitions from and generate lineagefor views and materialized views: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the Catalog Explorer , select the catalog you want to extract view definitions from and generate lineage for in Atlan. From the tabs at the top, click the Permissions tab, and then click Grant.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Select the user or service principal under Principals. Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Select the user or service principal under Principals.


Set up Databricks | Atlan Documentation

Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Click Grant to apply the permissions. Repeat steps 3 6 for each catalog you want to crawl in Atlan.


Set up Databricks | Atlan Documentation

SELECT permission is required to extract the definitions of views and materialized views. If you prefer not to grant SELECT at the catalog level, you can grant it on individual views and materialized views instead.


Set up Databricks | Atlan Documentation

(Optional) Grant permissions to mine query history To mine query history using REST API, you will need to assign the CAN MANAGE permission on your SQL warehouses to the user or service principal. To grant permissions to mine query history: Log in to your Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the left menu of your workspace, click SQL Warehouses. On the Compute page, for each SQL warehouse you want to mine query history, click the 3-dot icon and then click Permissions.


Set up Databricks | Atlan Documentation

Expand the Can use permissions dropdown and then select Can manage. This permission enables the service principal to view all queries for the warehouse.


Set up Databricks | Atlan Documentation

Click Add to assign the CAN MANAGE permission to the service principal. Personal access token authentication AWS service principal authentication Azure service principal authentication Grant permissions to crawl metadata (Optional) Grant permissions to query and preview data (Optional) Grant permissions to import and update tags (Optional) Grant permissions to extract lineage and usage from system tables (Optional) Grant view permissions to access Databricks entities via APIs (Optional) Grant permissions for views and materialized views (Optional) Grant permissions to mine query history


Set up Databricks | Atlan Documentation

Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods: Personal access token authentication AWS service principal authentication Azure service principal authentication Personal access token authentication Check that you have Admin and Databricks SQL access for the Databricks workspace.


Set up Databricks | Atlan Documentation

This is required for both cluster options described below. If you don't have this access, contact your Databricks administrator.


Set up Databricks | Atlan Documentation

Grant user access to workspace To grant workspace access to the user creating a personal access token: From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the user. From the tabs along the top of your workspace page, click the Permissions tab.


Set up Databricks | Atlan Documentation

In the upper right of the Permissions page, click Add permissions. In the Add permissions dialog, enter the following details: For User, group, or service principal , select the user to grant access.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the user to grant access.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. Generate a personal access token You can generate a personal access token in your Databricks workspace to the authenticate the integration in Atlan.


Set up Databricks | Atlan Documentation

To generate a personal access token: From the top right of your Databricks workspace, click your Databricks username, and then from the dropdown, click User Settings. Under the Settings menu, click Developer.


Set up Databricks | Atlan Documentation

On the Developer page, next to Access tokens , click Manage. On the Access tokens page, click the Generate new token button.


Set up Databricks | Atlan Documentation

In the Generate new token dialog: For Comment , enter a description of the token's intended use - for example, Atlan crawler. For Lifetime (days) , consider removing the number.


Set up Databricks | Atlan Documentation

This enables the token to be used indefinitely - it won't need to be refreshed. Important!


Set up Databricks | Atlan Documentation

If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate.


Set up Databricks | Atlan Documentation

For Comment , enter a description of the token's intended use - for example, Atlan crawler. For Comment , enter a description of the token's intended use - for example, Atlan crawler.


Set up Databricks | Atlan Documentation

For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed.


Set up Databricks | Atlan Documentation

Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

At the bottom of the dialog, click Generate. Copy and save the generated token in a secure location, and then click Done.


Set up Databricks | Atlan Documentation

Select a cluster Atlan recommends using serverless SQL warehouses for instant compute availability. To enable serverless SQL warehouses, refer to Databricks documentation for AWS Databricks workspaces or Microsoft documentation for Azure Databricks workspaces.


Set up Databricks | Atlan Documentation

You can set up personal access token authentication for your Databricks instance using one of the following cluster options: SQL warehouse (formerly SQL endpoint) To confirm an all-purpose interactive cluster is configured: From the left menu of any page of your Databricks instance, click Compute. Under the All-purpose clusters tab, verify you have a cluster defined.


Set up Databricks | Atlan Documentation

Click the link under the Name column of the table to open your cluster. Under the Configuration tab, verify the Autopilot options to Terminate after.


Set up Databricks | Atlan Documentation

minutes is enabled. At the bottom of the Configuration tab, expand the Advanced options expandable.


Set up Databricks | Atlan Documentation

Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path.


Set up Databricks | Atlan Documentation

To confirm a SQL warehouse is configured: From the left menu of any page of your Databricks instance, open the dropdown just below the databricks logo and change to SQL. From the refreshed left menu, click SQL Warehouses.


Set up Databricks | Atlan Documentation

Click the link under the Name column of the table to open your SQL warehouse. Under the Connection details tab, confirm that all of the fields are populated and copy them for use in crawling: Server hostname , Port , and HTTP path.


Set up Databricks | Atlan Documentation

AWS service principal authentication You need your AWS Databricks account admin to create a service principal and manage OAuth credentials for the service principal and your AWS Databricks workspace admin to add the service principal to your AWS Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Create a service principal You can create a service principal directly in your Databricks account or from a Databricks workspace.


Set up Databricks | Atlan Documentation

Identity federation enabled on your workspaces: Databricks recommends creating the service principal in the account and assigning it to workspaces. Identity federation disabled on your workspaces: Databricks recommends that you create your service principal from a workspace.


Set up Databricks | Atlan Documentation

To create a service principal from your Databricks account, with identify federation enabled: Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management.


Set up Databricks | Atlan Documentation

From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal.


Set up Databricks | Atlan Documentation

On the Add service principal page, enter a name for the service principal and then click Add. Once the service principal has been created, you can assign it to your identity federated workspace.


Set up Databricks | Atlan Documentation

From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab.


Set up Databricks | Atlan Documentation

In the upper right of the Permissions page, click Add permissions. In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the service principal you created.


Set up Databricks | Atlan Documentation

For Permission , click the dropdown and select workspace User. To create a service principal from a Databricks workspace, with identity federation disabled: Log in to your AWS Databricks workspace as a workspace admin.


Set up Databricks | Atlan Documentation

From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings. In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access.


Set up Databricks | Atlan Documentation

On the Identity and access page, under Management and permissions , next to Service principals , click Manage. In the upper right of the Service principals page, click Add service principal.


Set up Databricks | Atlan Documentation

In the Add service principal dialog, click the Add new button. For New service principal display name , enter a name for the service principal and then click Add.


Set up Databricks | Atlan Documentation

Create an OAuth secret for the service principal You need to create an OAuth secret to authenticate to Databricks REST APIs. To create an OAuth secret for the service principal : Log in to your Databricks account console as an account admin.


Set up Databricks | Atlan Documentation

Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management.


Set up Databricks | Atlan Documentation

From the left menu of the account console, click User management. From the tabs along the top of the User management page, click the Service principals tab.


Set up Databricks | Atlan Documentation

From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, select the service principal you created.


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, select the service principal you created. On the service principal page, under OAuth secrets , click Generate secret.


Set up Databricks | Atlan Documentation

On the service principal page, under OAuth secrets , click Generate secret. From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location.


Set up Databricks | Atlan Documentation

danger Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal.


Set up Databricks | Atlan Documentation

From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. Note that this secret is only revealed once during creation.


Set up Databricks | Atlan Documentation

The client ID is the same as the application ID of the service principal. Once you've copied the client ID and secret, click Done.


Set up Databricks | Atlan Documentation

Once you've copied the client ID and secret, click Done. Azure service principal authentication You need your Azure Databricks account admin to create a service principal and your Azure Databricks workspace admin to add the service principal to your Azure Databricks workspace - you may not have access yourself.


Set up Databricks | Atlan Documentation

You need the following to authenticate the connection in Atlan: Client ID (application ID) Tenant ID (directory ID) Create a service principal To use service principals on Azure Databricks , an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application and then add it to the Azure Databricks workspace to use as a service principal. To create a service principal: Sign in to the Azure portal.


Set up Databricks | Atlan Documentation

If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to create the service principal. Click**+ Add and select App registration**.


Set up Databricks | Atlan Documentation

For_Name_, enter a name for the application. In the_Supported account types_section, select Accounts in this organizational directory only (Single tenant) and then click Register.


Set up Databricks | Atlan Documentation

Application (client) ID Directory (tenant) ID To generate a client secret, within_Manage_, click Certificates & secrets. On the_Client secrets_tab, click New client secret.


Set up Databricks | Atlan Documentation

In the_Add a client secret_dialog, enter the following details: For Description , enter a description for the client secret. For_Expires_, select an expiry time period for the client secret and then click Add.


Set up Databricks | Atlan Documentation

Copy and store the client secret's_Value_in a secure place. Add a service principal to your account To add a service principal to your Azure Databricks account: Log in to your Azure Databricks account console as an account admin.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, click Add service principal. On the Add service principal page, enter a name for the service principal.


Set up Databricks | Atlan Documentation

Under UUID , paste the Application (client) ID for the service principal. Click Add.


Set up Databricks | Atlan Documentation

Assign a service principal to a workspace To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page.


Set up Databricks | Atlan Documentation

To assign a service principal to your Azure Databricks account: Log in to your Databricks account console as an account admin. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal.


Set up Databricks | Atlan Documentation

From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions.


Set up Databricks | Atlan Documentation

In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User.


Set up Databricks | Atlan Documentation

For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User.


Set up Databricks | Atlan Documentation

To assign a service principal to your Azure Databricks workspace: Log in to your Azure Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings.


Set up Databricks | Atlan Documentation

In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access. On the Identity and access page, under Management and permissions , next to Service principals , click Manage.


Set up Databricks | Atlan Documentation

In the upper right of the Service principals page, click Add service principal. In the Add service principal dialog, click the Add new button.


Set up Databricks | Atlan Documentation

For New service principal display name , paste the Application (client) ID for the service principal , enter a display name, and then click Add. Grant permissions to crawl metadata You must have a Unity Catalog-enabled Databricks workspace to crawl metadata in Atlan.


Set up Databricks | Atlan Documentation

To extract metadata, you can grant the BROWSE privilege , currently in public preview. You no longer require the Data Reader preset that granted the following privileges on objects in the catalog - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT.


Set up Databricks | Atlan Documentation

To grant permissions to a user or service principal: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the left menu of the Catalog Explorer page, select the catalog you want to crawl in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege.


Set up Databricks | Atlan Documentation

At the bottom of the dialog, click Grant. Under Principals , click the dropdown and then select the user or service principal.


Set up Databricks | Atlan Documentation

Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant.


Set up Databricks | Atlan Documentation

(Optional) Repeat steps 3-5 for each catalog you want to crawl in Atlan. System tables extraction method To crawl metadata via system tables, you must have a Unity Catalog-enabled workspace and a configured SQL warehouse.


Set up Databricks | Atlan Documentation

Follow these steps to extract metadata using system tables: Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Cross-workspace extraction To crawl metadata from all workspaces within a Databricks metastore using a single connection, see Set up cross-workspace extraction for instructions. (Optional) Grant permissions to query and preview data Atlan currently only supports querying data and viewing sample data preview for the personal access token authentication method.


Set up Databricks | Atlan Documentation

To grant permissions to query data and preview example data: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog.


Set up Databricks | Atlan Documentation

In the left menu of the Catalog Explorer page, select the catalog you want to query and preview data from in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button.


Set up Databricks | Atlan Documentation

In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog.


Set up Databricks | Atlan Documentation

Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT. At the bottom of the dialog, click Grant.


Set up Databricks | Atlan Documentation

Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog.


Set up Databricks | Atlan Documentation


Set up Databricks | Atlan Documentation

(Optional) Repeat steps 3-5 for each catalog you want to query and preview data from in Atlan. (Optional) Grant permissions to import and update tags To import Databricks tags , you must have a Unity Catalog-enabled workspace and a SQL warehouse configured.


Set up Databricks | Atlan Documentation

Atlan supports importing Databricks tags using system tables for all three authentication methods. Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following privileges: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalog_tags system.information_schema.schema_tags system.information_schema.table_tags system.information_schema.column_tags To push tags updated for assets in Atlan to Databricks, you need to grant the following privileges : APPLY TAG on the object USE CATALOG on the object's parent catalog USE SCHEMA on the object's parent schema (Optional) Grant permissions to extract lineage and usage from system tables You must have a Unity Catalog-enabled workspace to use system tables.