Crawl Databricks | Atlan Documentation

Once you have configured the Databricks access permissions , you can establish a connection between Atlan and your Databricks instance. (If you are also using AWS PrivateLink or Azure Private Link for Databricks, you will need to set that up first, too.) To crawl metadata from your Databricks instance, review the order of operations and then complete the following steps. Select the source To select Databricks as your source: In the top right corner of any screen, navigate to New and then click New Workflow . In the top right corner of any screen, navigate to New and then click New Workflow . From the list of packages, select Databricks Assets , and click Setup Workflow . From the list of packages, select Databricks Assets , and click Setup Workflow . Provide credentials Choose your extraction method: In Direct extraction, Atlan connects to your database and crawls metadata directly. Next, select an authentication method: In JDBC , you will need a personal access token and HTTP path for authentication . In AWS Service , you will need a client ID and client secret for AWS service principal authentication . In Azure Service , you will need a tenant ID, client ID, and client secret for Azure service principal authentication . In JDBC , you will need a personal access token and HTTP path for authentication . In AWS Service , you will need a client ID and client secret for AWS service principal authentication . In Azure Service , you will need a tenant ID, client ID, and client secret for Azure service principal authentication . In Offline extraction, you will need to first extract metadata yourself and make it available in S3 . In Agent extraction, Atlan's secure agent executes metadata extraction within the organization's environment. Direct extraction method To enter your Databricks credentials: For Host , enter the hostname, AWS PrivateLink endpoint , or Azure Private Link endpoint for your Databricks instance. For Port , enter the port number of your Databricks instance. For Personal Access Token , enter the access token you generated when setting up access . For HTTP Path , enter one of the following: A path starting with /sql/1.0/warehouses to use the Databricks SQL warehouse . A path starting with sql/protocolv1/o to use the Databricks interactive cluster . A path starting with /sql/1.0/warehouses to use the Databricks SQL warehouse . A path starting with sql/protocolv1/o to use the Databricks interactive cluster . Click Test Authentication to confirm connectivity to Databricks using these details. Once successful, at the bottom of the screen click Next . Make sure your Databricks instance (SQL warehouse or interactive cluster) is up and running, otherwise the Test Authentication step times out. To enter your Databricks credentials: For Host , enter the hostname or AWS PrivateLink endpoint for your Databricks instance. For Port , enter the port number of your Databricks instance. For Client ID , enter the client ID for your AWS service principal . For Client Secret , enter the client secret for your AWS service principal . Click Test Authentication to confirm connectivity to Databricks using these details. Once successful, at the bottom of the screen click Next . To enter your Databricks credentials: For Host , enter the hostname or Azure Private Link endpoint for your Databricks instance. For Port , enter the port number of your Databricks instance. For Client ID , enter the application (client) ID for your Azure service principal . For Client Secret , enter the client secret for your Azure service principal . For Tenant ID , enter the directory (tenant) ID for your Azure service principal . Click Test Authentication to confirm connectivity to Databricks using these details. Once successful, at the bottom of the screen click Next . Offline extraction method Atlan supports the offline extraction method for fetching metadata from Databricks. This method uses Atlan's databricks-extractor tool to fetch metadata. You need to first extract the metadata yourself and then make it available in S3 . To enter your S3 details: For Bucket name , enter the name of your S3 bucket. For Bucket prefix , enter the S3 prefix under which all the metadata files exist. These include output/databricks-example/catalogs/success/result-0.json , output/databricks-example/schemas/{{catalog_name}}/success/result-0.json , output/databricks-example/tables/{{catalog_name}}/success/result-0.json , and similar files. (Optional) For Bucket region , enter the name of the S3 region. When complete, at the bottom of the screen, click Next . Agent extraction method Atlan supports using a Secure Agent for fetching metadata from Databricks. To use a Secure Agent, follow these steps: Select the Agent tab. Configure the Databricks data source by adding the secret keys for your secret store. For details on the required fields, refer to the Direct extraction section. Complete the Secure Agent configuration by following the instructions in the How to configure Secure Agent for workflow execution guide. Click Next after completing the configuration. Configure the connection To complete the Databricks connection configuration: Provide a Connection Name that represents your source environment. For example, you might want to use values like production , development , gold , or analytics . Provide a Connection Name that represents your source environment. For example, you might want to use values like production , development , gold , or analytics . (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins . danger If you don't specify any user or group, nobody can manage the connection - not even admins. (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins . If you don't specify any user or group, nobody can manage the connection - not even admins. (Optional) To prevent users from querying any Databricks data, change Enable SQL Query to No . (Optional) To prevent users from querying any Databricks data, change Enable SQL Query to No . (Optional) To prevent users from previewing any Databricks data, change Enable Data Preview to No . (Optional) To prevent users from previewing any Databricks data, change Enable Data Preview to No . (Optional) To prevent users from running large queries, change Max Row Limit or keep the default selection. (Optional) To prevent users from running large queries, change Max Row Limit or keep the default selection. At the bottom of the screen, click the Next button to proceed. At the bottom of the screen, click the Next button to proceed. Configure the crawler Before running the Databricks crawler, you can further configure it. System tables extraction method The system metadata extraction method is only available for Unity Catalog-enabled workspaces . It provides access to detailed metadata from system tables and supports all three authentication types. You can extract metadata from your Databricks workspace using this method. Follow these steps: Set up authentication using one of the following: Personal access token AWS service principal Azure service principal Set up authentication using one of the following: Personal access token AWS service principal Azure service principal The default options can work as is. You may choose to override the defaults for any of the remaining options: For Asset selection , select a filtering option: For SQL warehouse , click the dropdown to select the SQL warehouse you want to configure. To select the assets you want to include in crawling, click Include by hierarchy and filter for assets down to the database or schema level. (This defaults to all assets, if none are specified.) To have the crawler include Databases , Schemas , or Tables & Views based on a naming convention, click Include by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_DB.* for Databases includes all the matching databases and their child assets. To select the assets you want to exclude from crawling, click Exclude by hierarchy and filter for assets down to the database or schema level. (This defaults to no assets, if none are specified.) To have the crawler ignore Databases , Schemas , or Tables & Views based on a naming convention, click Exclude by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_TABLES.* for Tables & Views excludes all the matching tables and views. Click + to add more filters. If you add multiple filters, assets are crawled based on matching all the filtering conditions you have set. To import tags from Databricks to Atlan , change Import Tags to Yes . Note that you must have a Unity Catalog-enabled workspace to import Databricks tags in Atlan. Did you know? If an asset appears in both the include and exclude filters, the exclude filter takes precedence. The default options can work as is. You may choose to override the defaults for any of the remaining options: For Asset selection , select a filtering option: For SQL warehouse , click the dropdown to select the SQL warehouse you want to configure. To select the assets you want to include in crawling, click Include by hierarchy and filter for assets down to the database or schema level. (This defaults to all assets, if none are specified.) To have the crawler include Databases , Schemas , or Tables & Views based on a naming convention, click Include by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_DB.* for Databases includes all the matching databases and their child assets. To select the assets you want to exclude from crawling, click Exclude by hierarchy and filter for assets down to the database or schema level. (This defaults to no assets, if none are specified.) To have the crawler ignore Databases , Schemas , or Tables & Views based on a naming convention, click Exclude by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_TABLES.* for Tables & Views excludes all the matching tables and views. Click + to add more filters. If you add multiple filters, assets are crawled based on matching all the filtering conditions you have set. To import tags from Databricks to Atlan , change Import Tags to Yes . Note that you must have a Unity Catalog-enabled workspace to import Databricks tags in Atlan. Did you know? If an asset appears in both the include and exclude filters, the exclude filter takes precedence. For Asset selection , select a filtering option: For SQL warehouse , click the dropdown to select the SQL warehouse you want to configure. To select the assets you want to include in crawling, click Include by hierarchy and filter for assets down to the database or schema level. (This defaults to all assets, if none are specified.) To have the crawler include Databases , Schemas , or Tables & Views based on a naming convention, click Include by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_DB.* for Databases includes all the matching databases and their child assets. To select the assets you want to exclude from crawling, click Exclude by hierarchy and filter for assets down to the database or schema level. (This defaults to no assets, if none are specified.) To have the crawler ignore Databases , Schemas , or Tables & Views based on a naming convention, click Exclude by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_TABLES.* for Tables & Views excludes all the matching tables and views. Click + to add more filters. If you add multiple filters, assets are crawled based on matching all the filtering conditions you have set. To import tags from Databricks to Atlan , change Import Tags to Yes . Note that you must have a Unity Catalog-enabled workspace to import Databricks tags in Atlan. If an asset appears in both the include and exclude filters, the exclude filter takes precedence. Toggle incremental extraction, for a faster and more efficient metadata extraction. JDBC extraction method The JDBC extraction method uses JDBC queries to extract metadata from your Databricks instance. This was the original extraction method provided by Databricks. This extraction method is only supported for personal access token authentication . You can override the defaults for any of these options: To select the assets you want to include in crawling, click Include Metadata . (This will default to all assets, if none are specified.) To select the assets you want to exclude from crawling, click Exclude Metadata . (This will default to no assets if none are specified.) To have the crawler ignore tables and views based on a naming convention, specify a regular expression in the Exclude regex for tables & views field. For View Definition Lineage , keep the default Yes to generate upstream lineage for views based on the tables referenced in the views or click No to exclude from crawling. For Advanced Config , keep Default for the default configuration or click Advanced to further configure the crawler: To enable or disable schema-level filtering at source, click Enable Source Level Filtering and select True to enable it or False to disable it. To enable or disable schema-level filtering at source, click Enable Source Level Filtering and select True to enable it or False to disable it. REST API extraction method The REST API extraction method uses Unity Catalog to extract metadata from your Databricks instance. This extraction method is supported for all three authentication options: personal access token , AWS service principal , and Azure service principal . This method is only supported by Unity Catalog-enabled workspaces. If you enable an existing workspace, you also need to upgrade your tables and views to Unity Catalog . While REST APIs are used to extract metadata, JDBC queries are still used for querying purposes. You can override the defaults for any of these options: Change the extraction method under Extraction method to REST API . To select the assets you want to include in crawling, click Include Metadata . (This will default to all assets, if none are specified.) To select the assets you want to exclude from crawling, click Exclude Metadata . (This will default to no assets if none are specified.) To import tags from Databricks to Atlan , change Import Tags to Yes . Note that you must have a Unity Catalog-enabled workspace to import Databricks tags in Atlan. For SQL warehouse , click the dropdown to select the SQL warehouse you have configured. For SQL warehouse , click the dropdown to select the SQL warehouse you have configured. If an asset appears in both the include and exclude filters, the exclude filter takes precedence. Run the crawler Follow these steps to run the Databricks crawler: To check for any permissions or other configuration issues before running the crawler, click Preflight checks . You can either: To run the crawler once immediately, at the bottom of the screen, click the Run button. To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule Run button. To run the crawler once immediately, at the bottom of the screen, click the Run button. To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule Run button. Once the crawler has completed running, you will see the assets in Atlan's asset page! ð Select the source Configure the connection Configure the crawler Run the crawler


Set up Databricks | Atlan Documentation

Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods: Personal access token authentication AWS service principal authentication Azure service principal authentication Personal access token authentication Check that you have Admin and Databricks SQL access for the Databricks workspace. This is required for both cluster options described below. If you don't have this access, contact your Databricks administrator. Grant user access to workspace To grant workspace access to the user creating a personal access token: From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the user. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the user to grant access. For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the user to grant access. For Permission , click the dropdown and select workspace User. Generate a personal access token You can generate a personal access token in your Databricks workspace to the authenticate the integration in Atlan . To generate a personal access token: From the top right of your Databricks workspace, click your Databricks username, and then from the dropdown, click User Settings . Under the Settings menu, click Developer . On the Developer page, next to Access tokens , click Manage . On the Access tokens page, click the Generate new token button. In the Generate new token dialog: For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate . For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate . At the bottom of the dialog, click Generate . Copy and save the generated token in a secure location, and then click Done . Select a cluster Atlan recommends using serverless SQL warehouses for instant compute availability. To enable serverless SQL warehouses, refer to Databricks documentation for AWS Databricks workspaces or Microsoft documentation for Azure Databricks workspaces. You can set up personal access token authentication for your Databricks instance using one of the following cluster options: SQL warehouse (formerly SQL endpoint) To confirm an all-purpose interactive cluster is configured: From the left menu of any page of your Databricks instance, click Compute . Under the All-purpose clusters tab, verify you have a cluster defined. Click the link under the Name column of the table to open your cluster. Under the Configuration tab, verify the Autopilot options to Terminate after . minutes is enabled. At the bottom of the Configuration tab, expand the Advanced options expandable. Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path . Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path . To confirm a SQL warehouse is configured: From the left menu of any page of your Databricks instance, open the dropdown just below the databricks logo and change to SQL . From the refreshed left menu, click SQL Warehouses . Click the link under the Name column of the table to open your SQL warehouse. Under the Connection details tab, confirm that all of the fields are populated and copy them for use in crawling: Server hostname , Port , and HTTP path . AWS service principal authentication You need your AWS Databricks account admin to create a service principal and manage OAuth credentials for the service principal and your AWS Databricks workspace admin to add the service principal to your AWS Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Create a service principal You can create a service principal directly in your Databricks account or from a Databricks workspace. Identity federation enabled on your workspaces: Databricks recommends creating the service principal in the account and assigning it to workspaces. Identity federation disabled on your workspaces: Databricks recommends that you create your service principal from a workspace. To create a service principal from your Databricks account, with identify federation enabled: Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal . On the Add service principal page, enter a name for the service principal and then click Add . Once the service principal has been created, you can assign it to your identity federated workspace. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the service principal you created. For Permission , click the dropdown and select workspace User. To create a service principal from a Databricks workspace, with identity federation disabled: Log in to your AWS Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings . In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access . On the Identity and access page, under Management and permissions , next to Service principals , click Manage . In the upper right of the Service principals page, click Add service principal . In the Add service principal dialog, click the Add new button. For New service principal display name , enter a name for the service principal and then click Add . Create an OAuth secret for the service principal You need to create an OAuth secret to authenticate to Databricks REST APIs. To create an OAuth secret for the service principal : Log in to your Databricks account console as an account admin. Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management . From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, select the service principal you created . In the upper right of the Service principals page, select the service principal you created . On the service principal page, under OAuth secrets , click Generate secret . On the service principal page, under OAuth secrets , click Generate secret . From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. danger Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal. From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal. Once you've copied the client ID and secret, click Done . Once you've copied the client ID and secret, click Done . Azure service principal authentication You need your Azure Databricks account admin to create a service principal and your Azure Databricks workspace admin to add the service principal to your Azure Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Client ID (application ID) Tenant ID (directory ID) Create a service principal To use service principals on Azure Databricks , an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application and then add it to the Azure Databricks workspace to use as a service principal. To create a service principal: Sign in to the Azure portal . If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to create the service principal. Click**+ Add and select App registration**. For_Name_, enter a name for the application. In the_Supported account types_section, select Accounts in this organizational directory only (Single tenant) and then click Register . Application (client) ID Directory (tenant) ID To generate a client secret, within_Manage_, click Certificates & secrets . On the_Client secrets_tab, click New client secret . In the_Add a client secret_dialog, enter the following details: For Description , enter a description for the client secret. For_Expires_, select an expiry time period for the client secret and then click Add . Copy and store the client secret's_Value_in a secure place. Add a service principal to your account To add a service principal to your Azure Databricks account: Log in to your Azure Databricks account console as an account admin. From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal . On the Add service principal page, enter a name for the service principal. Under UUID , paste the Application (client) ID for the service principal. Click Add . Assign a service principal to a workspace To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page. To assign a service principal to your Azure Databricks account: Log in to your Databricks account console as an account admin. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User . For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User . To assign a service principal to your Azure Databricks workspace: Log in to your Azure Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings . In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access . On the Identity and access page, under Management and permissions , next to Service principals , click Manage . In the upper right of the Service principals page, click Add service principal . In the Add service principal dialog, click the Add new button. For New service principal display name , paste the Application (client) ID for the service principal , enter a display name, and then click Add . Grant permissions to crawl metadata You must have a Unity Catalog-enabled Databricks workspace to crawl metadata in Atlan. To extract metadata, you can grant the BROWSE privilege , currently in public preview. You no longer require the Data Reader preset that granted the following privileges on objects in the catalog - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . To grant permissions to a user or service principal: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the left menu of the Catalog Explorer page, select the catalog you want to crawl in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button. In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant . Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant . (Optional) Repeat steps 3-5 for each catalog you want to crawl in Atlan. System tables extraction method To crawl metadata via system tables, you must have a Unity Catalog-enabled workspace and a configured SQL warehouse. Follow these steps to extract metadata using system tables: Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Cross-workspace extraction To crawl metadata from all workspaces within a Databricks metastore using a single connection, see Set up cross-workspace extraction for instructions. (Optional) Grant permissions to query and preview data Atlan currently only supports querying data and viewing sample data preview for the personal access token authentication method. To grant permissions to query data and preview example data: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the left menu of the Catalog Explorer page, select the catalog you want to query and preview data from in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button. In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog. Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . At the bottom of the dialog, click Grant . Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog. Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . At the bottom of the dialog, click Grant . (Optional) Repeat steps 3-5 for each catalog you want to query and preview data from in Atlan. (Optional) Grant permissions to import and update tags To import Databricks tags , you must have a Unity Catalog-enabled workspace and a SQL warehouse configured. Atlan supports importing Databricks tags using system tables for all three authentication methods. Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following privileges: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalog_tags system.information_schema.schema_tags system.information_schema.table_tags system.information_schema.column_tags To push tags updated for assets in Atlan to Databricks, you need to grant the following privileges : APPLY TAG on the object USE CATALOG on the object's parent catalog USE SCHEMA on the object's parent schema (Optional) Grant permissions to extract lineage and usage from system tables You must have a Unity Catalog-enabled workspace to use system tables. Atlan supports extracting the following for your Databricks assets using system tables : usage and popularity metrics Enable system.access schema You need your account admin to enable the system.access schema using the SystemSchemas API . This enables Atlan to extract lineage using system tables. In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore in your Databricks environment for which you want to extract lineage. To verify that system schemas are enabled for each schema, follow the steps in Databricks documentation : List system schemas using the SystemSchemas API to check the status. If enabled for any given schema, the state is EnableCompleted . This confirms that the schema has been enabled for that specific metastore. Atlan can only extract lineage using system tables when the state is marked as EnableCompleted . (Optional) enable system.information_schema.table To generate lineage with the target type set as PATH for a table, Atlan uses metadata from system.information_schema.table to resolve table paths and dependencies. To enable this, you must grant the following permissions on the relevant catalog, schema, and tables. You must be a metastore admin, have the MANAGE privilege on the object, or be the owner of the catalog, schema, or table to grant these permissions. In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore from which you want to extract lineage. Open Catalog Explorer in your Databricks workspace. Navigate to the catalog (for example, main ) and then to the appropriate schema (for example, sales ). Click the Permissions tab. Click Grant . Enter the user or group name (principal). Assign the following permissions: USAGE on the catalog USAGE on the schema SELECT on each relevant table USAGE on the catalog USAGE on the schema SELECT on each relevant table Click Grant to apply the changes. These privileges enable Atlan to read table definitions and other metadata from the metastore. (Optional) enable system.query schema This is only required if you also want to extract usage and popularity metrics from Databricks. You need your account admin to enable the system.query schema using the SystemSchemas API . This enables Atlan to mine query history using system tables for usage and popularity metrics. To verify that system schemas is enabled for each schema, follow the steps in Databricks documentation . If enabled for any given schema, the state is EnableCompleted . ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query ? Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables . Grant permissions Atlan supports extracting Databricks lineage and usage and popularity metrics using system tables for all three authentication methods . Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following permissions: CAN_USE on a SQL warehouse USE_CATALOG on system catalog USE SCHEMA on system.access schema USE SCHEMA on system.query schema (tomine query history for usage and popularity metrics) SELECT on the following tables: system.query.history (to mine query history for usage and popularity metrics) system.access.table_lineage system.access.column_lineage system.query.history (to mine query history for usage and popularity metrics) You need to create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection. ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query ? Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables . (Optional) Create cloned views of system tables When you don't want to grant access to system tables directly, you can create cloned views to expose lineage and popularity metrics through a separate schema. Follow these steps to set up cloned views: Create a catalog and schema to store cloned views. Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema . Create a catalog and schema to store cloned views. Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema . Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW . . column_lineage AS SELECT * FROM system . access . column_lineage ; CREATE OR REPLACE VIEW . . table_lineage AS SELECT * FROM system . access . table_lineage ; Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW . . query_history AS SELECT * FROM system . query . history ; Replace and with the catalog and schema names used in your environment. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW . . column_lineage AS SELECT * FROM system . access . column_lineage ; CREATE OR REPLACE VIEW . . table_lineage AS SELECT * FROM system . access . table_lineage ; Replace and with the catalog and schema names used in your environment. Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW . . query_history AS SELECT * FROM system . query . history ; Replace and with the catalog and schema names used in your environment. Replace and with the catalog and schema names used in your environment. Grant the following permissions to enable access to the cloned views: CAN_USE on a SQL warehouse USE CATALOG on the catalog (for example, ) USE SCHEMA and SELECT on the schema (for example, . ) You must create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection. Locate warehouse ID To extract lineage and usage and popularity metrics using system tables, you will also need the warehouse ID of your SQL warehouse . To locate the warehouse ID: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click SQL Warehouses . On the Compute page, select the warehouse you want to use. (Optional) Grant view permissions to access Databricks entities via APIs Atlan uses Databricks REST APIs to extract metadata for Notebooks, Queries, Jobs, and Pipelines. This information helps to understand which Databricks enitity was used to create a lineage between assets. Use the steps below for each object type to grant CAN VIEW permission to the Databricks user or service principal configured in your integration: Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders . Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders . Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries . Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries . Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job . Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job . Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions . Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions . (Optional) Grant permissions for views and materialized views Atlan requires the following permissions to to extract view definitions from and generate lineagefor views and materialized views: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the Catalog Explorer , select the catalog you want to extract view definitions from and generate lineage for in Atlan. From the tabs at the top, click the Permissions tab, and then click Grant . In the Grant on (workspace name) dialog, configure the following: Select the user or service principal under Principals . Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Select the user or service principal under Principals . Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Click Grant to apply the permissions. Repeat steps 3 6 for each catalog you want to crawl in Atlan. SELECT permission is required to extract the definitions of views and materialized views. If you prefer not to grant SELECT at the catalog level, you can grant it on individual views and materialized views instead. (Optional) Grant permissions to mine query history To mine query history using REST API, you will need to assign the CAN MANAGE permission on your SQL warehouses to the user or service principal. To grant permissions to mine query history: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click SQL Warehouses . On the Compute page, for each SQL warehouse you want to mine query history, click the 3-dot icon and then click Permissions . Expand the Can use permissions dropdown and then select Can manage . This permission enables the service principal to view all queries for the warehouse . Click Add to assign the CAN MANAGE permission to the service principal. Personal access token authentication AWS service principal authentication Azure service principal authentication Grant permissions to crawl metadata (Optional) Grant permissions to query and preview data (Optional) Grant permissions to import and update tags (Optional) Grant permissions to extract lineage and usage from system tables (Optional) Grant view permissions to access Databricks entities via APIs (Optional) Grant permissions for views and materialized views (Optional) Grant permissions to mine query history


Set up Databricks | Atlan Documentation


Set up cross-workspace extraction | Atlan Documentation

Eliminate the need for separate crawler configurations by using a single service principal to crawl metadata from all workspaces within a Databricks metastore. This guide walks you through configuring the necessary permissions to enable cross-workspace extraction. Cross-workspace extraction isn't supported for REST API or JDBC extraction methods. Before you begin, make sure you have: A Unity Catalog-enabled Databricks workspace Account admin access to create and manage service principals Workspace admin access to grant permissions across all target workspaces At least one active SQL warehouse in each workspace you intend to crawl Set up Databricks authentication completed with one of the supported authentication methods System table extraction enabled for lineage and usage extraction Permissions required The service principal needs the following permissions to enable cross-workspace extraction: CAN_USE on SQL warehouses in each workspace SELECT on system.access.workspace_latest table USE CATALOG , BROWSE , and SELECT on all catalogs you want to crawl Add service principal to all workspaces You must use a single, common service principal that has been granted access to all Databricks workspaces you intend to crawl within the metastore. Log in to your Databricks account console as an account admin From the left menu, click Workspaces and select a workspace From the tabs along the top, click the Permissions tab In the upper right, click Add permissions In the Add permissions dialog: For User, group, or service principal , select your service principal For Permission , select workspace User Click Add For User, group, or service principal , select your service principal For Permission , select workspace User Repeat steps 2-5 for each workspace you intend to crawl Grant permissions Configure the necessary permissions for the service principal to access and extract metadata from all workspaces within the metastore. SQL workspace permissions: The service principal must have usage permissions on at least one active SQL warehouse within each workspace . The extractor uses the smallest available warehouse to run its discovery queries. Connect to your Databricks workspace using a SQL client or the SQL editor Connect to your Databricks workspace using a SQL client or the SQL editor Run the following command for each workspace, replacing the placeholders: GRANT CAN_USE ON WAREHOUSE TO ` ` ; Replace with your actual warehouse name Replace with your service principal's application ID Example GRANT CAN_USE ON WAREHOUSE production - warehouse TO ` 12345678-1234-1234-1234-123456789012 ` ; Run the following command for each workspace, replacing the placeholders: Replace with your actual warehouse name Replace with your service principal's application ID Log in to your Databricks workspace as a workspace admin From the left menu, click SQL Warehouses On the Compute page, for each SQL warehouse, click the 3-dot icon and then click Permissions Select Can use permission Click Add to assign the permission System table permissions: Access to the system schema is essential for workspace and lineage discovery. Via SQL Via UI Connect to your Databricks workspace using a SQL client or the SQL editor Run the following command, replacing the placeholder: GRANT SELECT ON TABLE system . access . workspace_latest TO ` ` ; Replace with your service principal's application ID Example GRANT SELECT ON TABLE system . access . workspace_latest TO ` 12345678-1234-1234-1234-123456789012 ` ; Log in to your Databricks workspace as a workspace admin From the left menu, click Catalog In the Catalog Explorer , navigate to system > access Click on the workspace_latest table Click the Permissions tab and then click Grant In the Grant permissions dialog: Under Principals , select your service principal Under Privileges , check SELECT Click Grant to apply the permissions System table permissions: Access to the system schema is essential for workspace and lineage discovery. Connect to your Databricks workspace using a SQL client or the SQL editor Connect to your Databricks workspace using a SQL client or the SQL editor Run the following command, replacing the placeholder: GRANT SELECT ON TABLE system . access . workspace_latest TO ` ` ; Replace with your service principal's application ID Example GRANT SELECT ON TABLE system . access . workspace_latest TO ` 12345678-1234-1234-1234-123456789012 ` ; Run the following command, replacing the placeholder: Replace with your service principal's application ID Log in to your Databricks workspace as a workspace admin From the left menu, click Catalog In the Catalog Explorer , navigate to system > access Click on the workspace_latest table Click the Permissions tab and then click Grant In the Grant permissions dialog: Under Principals , select your service principal Under Privileges , check SELECT Click Grant to apply the permissions Under Principals , select your service principal Under Privileges , check SELECT Click Grant to apply the permissions Asset permissions: The service principal requires permissions to "see" and "read" the metadata for all data assets you wish to extract. These grants must be applied to all private, public, and shared catalogs that are in scope for crawling. Important! For private catalogs, grant permissions from each workspace. For public catalogs, grant from any workspace. Via SQL Via UI Connect to your Databricks workspace using a SQL client or the SQL editor Grant catalog-level permissions (required even when using BROWSE - BROWSE automatically grants access to all schemas and tables): GRANT USE CATALOG ON CATALOG TO ` ` ; GRANT BROWSE ON CATALOG TO ` ` ; Replace with your actual catalog name Replace with your service principal's application ID If not using BROWSE, along with catalog permissions, grant additional permissions: Grant schema-level permissions: GRANT USE SCHEMA ON SCHEMA . TO ` ` ; Replace and with your actual values Replace with your service principal's application ID Example GRANT USE CATALOG ON CATALOG main TO ` 12345678-1234-1234-1234-123456789012 ` ; GRANT BROWSE ON CATALOG main TO ` 12345678-1234-1234-1234-123456789012 ` ; Log in to your Databricks workspace as a workspace admin From the left menu, click Catalog In the Catalog Explorer , navigate to the catalog you want to grant permissions on (for example, main ) Click the Permissions tab and then click Grant In the Grant permissions dialog: Under Principals , select your service principal Under Privileges , check the following permissions: USE CATALOG USE SCHEMA BROWSE SELECT Click Grant to apply the permissions Repeat steps 3-5 for each catalog you want to crawl in Atlan Asset permissions: The service principal requires permissions to "see" and "read" the metadata for all data assets you wish to extract. These grants must be applied to all private, public, and shared catalogs that are in scope for crawling. For private catalogs, grant permissions from each workspace. For public catalogs, grant from any workspace. Connect to your Databricks workspace using a SQL client or the SQL editor Connect to your Databricks workspace using a SQL client or the SQL editor Grant catalog-level permissions (required even when using BROWSE - BROWSE automatically grants access to all schemas and tables): GRANT USE CATALOG ON CATALOG TO ` ` ; GRANT BROWSE ON CATALOG TO ` ` ; Replace with your actual catalog name Replace with your service principal's application ID Grant catalog-level permissions (required even when using BROWSE - BROWSE automatically grants access to all schemas and tables): Replace with your actual catalog name Replace with your service principal's application ID If not using BROWSE, along with catalog permissions, grant additional permissions: Grant schema-level permissions: GRANT USE SCHEMA ON SCHEMA . TO ` ` ; Replace and with your actual values Replace with your service principal's application ID If not using BROWSE, along with catalog permissions, grant additional permissions: Grant schema-level permissions: GRANT USE SCHEMA ON SCHEMA . TO ` ` ; Replace and with your actual values Replace with your service principal's application ID Grant schema-level permissions: Replace and with your actual values Replace with your service principal's application ID Log in to your Databricks workspace as a workspace admin From the left menu, click Catalog In the Catalog Explorer , navigate to the catalog you want to grant permissions on (for example, main ) Click the Permissions tab and then click Grant In the Grant permissions dialog: Under Principals , select your service principal Under Privileges , check the following permissions: USE CATALOG USE SCHEMA BROWSE SELECT Click Grant to apply the permissions Under Principals , select your service principal Under Privileges , check the following permissions: Click Grant to apply the permissions Repeat steps 3-5 for each catalog you want to crawl in Atlan Need help? Check Troubleshooting Databricks connectivity for common issues Contact Atlan support for help with setup or integration Next steps Add service principal to all workspaces


query data | Atlan Documentation

There are two ways to query data in Atlan: writing your own SQL using the Visual Query Builder Atlan pushes all queries to the source (no data is stored in Atlan). In addition, Atlan applies access policies to the results before displaying them. Write your own SQL Anyone with the knowledge to write SQL. Any Atlan user with data access to the asset can query data. To query an asset with your own SQL: From the left menu of any screen, click Insights . Use the Select database dropdown to choose another database, if necessary. Hover over the table or view, and click the play icon. This writes and runs a basic preview query. (Optional) Click the open asset sidebar icon to view more details in the asset sidebar. (Optional) Click the eye icon to view a preview of the query results. (Optional) Click the 3-dot icon for more options: Click Set editor context to set the same connection, database, and schema name in the query editor as selected in the Explorer tab. Click Place name in editor to view the asset name in the query editor. Click Copy path to copy the full path of the asset, including database and schema names. (Optional) Click the open asset sidebar icon to view more details in the asset sidebar. (Optional) Click the eye icon to view a preview of the query results. (Optional) Click the 3-dot icon for more options: Click Set editor context to set the same connection, database, and schema name in the query editor as selected in the Explorer tab. Click Place name in editor to view the asset name in the query editor. Click Copy path to copy the full path of the asset, including database and schema names. Click Set editor context to set the same connection, database, and schema name in the query editor as selected in the Explorer tab. Click Place name in editor to view the asset name in the query editor. Click Copy path to copy the full path of the asset, including database and schema names. Under the Untitled tab on the right, change the sample query or write your own - separate multiple queries with a semicolon ; . Click the Run button in the upper right to test your query as you write it. (Optional) Click the downward arrow next to the Run button to export query results via email or schedule the query . (Optional) Click the downward arrow next to the Run button to export query results via email or schedule the query . (Optional) If you have multiple tabs open in the query editor, right-click a tab to open the tabs menu. You can close a specific tab or all tabs, or duplicate the query. (Optional) From the top right of the query editor, click the 3-dot icon for additional query editor actions or to customize it further: Click or hover over Duplicate query to create a duplicate version of your query. Click or hover over Open command palette to view the actions you can run inside the query editor. Click or hover over Themes and then select your preferred theme for the query editor. Click or hover over Tab spacing to change the tab spacing for your queries. Click or hover over Font size to change the font size for your queries. Click or hover over Cursor to change the cursor position in the query editor. Click or hover over Autosuggestions to turn off autosuggestions for assets in the query editor. Click or hover over Duplicate query to create a duplicate version of your query. Click or hover over Open command palette to view the actions you can run inside the query editor. Click or hover over Themes and then select your preferred theme for the query editor. Click or hover over Tab spacing to change the tab spacing for your queries. Click or hover over Font size to change the font size for your queries. Click or hover over Cursor to change the cursor position in the query editor. Click or hover over Autosuggestions to turn off autosuggestions for assets in the query editor. The editor supports all read-based SQL statements, including JOIN . The editor will not run any write-based statements. The following SQL statements are not supported: You can select the context for your query to the left of the Run button. Then you won't need to fully qualify table names with schema and database names. Use the Visual Query Builder Any Atlan user with data access to the asset . No SQL knowledge required! To query an asset using the Visual Query Builder: From the left menu of any screen, click Insights . At the top of the screen, to the right of the Untitled tab, click the + button and select New visual query . Under Select from choose the table or view you want to query. (Optional) In the column selector to the right, select the column you want to query. Then develop your query: Click the Run button to run the query and preview its results. Click the blue circular + button to add an action to the query. Repeat these steps until your query is complete. Click the Run button to run the query and preview its results. Click the blue circular + button to add an action to the query. Repeat these steps until your query is complete. (Optional) If there are any errors in your query, click Auto fix for Atlan to recommend a fix. (Optional) In the query results set, click Copy to copy the query results or click Download to export them. You can learn more about the query builder actions in this example . Write your own SQL Use the Visual Query Builder


Provide credentials to view sample data | Atlan Documentation

Once your connection admins have configured bring your own credentials (BYOC) in Atlan, users will need to provide their own credentials before they can view the sample data in the asset profile. This will help you enforce better governance across your organization. Any Atlan user with data access to the asset and their own credentials for the data store. Atlan will display a 100-row sample of the data . Use your own credentials to view sample data Atlan supports both basic username and password as well as key pair authentication of your credentials. Atlan also supports SSO authentication . To set up your own credentials for viewing sample data: On the Assets page, click on an asset to view its asset profile. In the asset profile, click Sample Data . To set up your credentials for viewing the sample data, click Get Started . In the popup window, click Get Started once again to proceed. In the User credential setup dialog box, Basic is selected as the default authentication option. Enter the following: For Username , enter the username for the connection. For Password , enter the password for that connection. For Role , enter your role for that connection. For Warehouse , enter the name of the warehouse. For Username , enter the username for the connection. For Password , enter the password for that connection. For Role , enter your role for that connection. For Warehouse , enter the name of the warehouse. Click the Test Authentication button to confirm your credentials. Once authentication is successful, click Done . You can now view sample data using your own credentials! ð When using the key pair method, you'll need to enter your encrypted private key and the private key password to complete the authentication process. Once you've set up your credentials for viewing sample data, you can also manage your credentials . If your admin has enabled sample data download , you can export sample data in a CSV file. Use your own credentials to view sample data


Manage Databricks tags | Atlan Documentation

You must have a Unity Catalog-enabled workspace to import Databricks tags in Atlan. Atlan enables you to import your Databricks tags , update your Databricks assets with the imported tags, and push the tag updates back to Databricks: Import tags - crawl Databricks tags from Databricks to Atlan Reverse sync - sync Databricks tag updates from Atlan to Databricks Once you've imported your Databricks tags to Atlan: Your Databricks assets in Atlan will be automatically enriched with their Databricks tags. Imported Databricks tags will be mapped to corresponding Atlan tags through case-insensitive name match - multiple Databricks tags can be matched to a single tag in Atlan. You can also attach Databricks tags , including tag values, to your Databricks assets in Atlan - allowing you to categorize your assets at a more granular level. You can filter your assets by Databricks tags and tag values. You can enable reverse sync to push any tag updates for your Databricks assets back to Databricks - including tag values added to assets in Atlan. Enabling reverse sync will only update existing tags in Databricks. It will neither create nor delete any tags in Databricks. You must have a Unity Catalog-enabled workspace and SQL warehouse configured to import Databricks tags in Atlan. Before you can import tags from and push tag updates to Databricks using personal access token , AWS service principal , or Azure service principal authentication, you will need to do the following: Ensure that you have a Unity Catalog-enabled workspace and a SQL warehouse configured. Create tags or have existing tags in Databricks. Grant permissions to import tags from and push tag updates to Databricks. Import Databricks tags to Atlan You will need to be an admin user in Atlan to import Databricks tags to Atlan. You will also need to work with your Databricks administrator to grant permissions to import tags from Databricks - you may not have access yourself. You can import your Databricks tags to Atlan through one-way tag sync. The synced Databricks tags will be matched to corresponding tags in Atlan through case-insensitive name match and your Databricks assets will be enriched with their synced tags from Databricks. To import Databricks tags to Atlan, you can either: Create a new Databricks workflow and configure the crawler to import tags. Modify the crawler's configuration for an existing Databricks workflow to change Import Tags to Yes . If you subsequently modify the workflow to disable tag import, for any tags already imported, Atlan will preserve those tags. Once the crawler has completed running, tags imported from Databricks will be available to use for tagging assets ! ð View Databricks tags in Atlan Once you've imported your Databricks tags, you will be able to view and manage your Databricks tags in Atlan. To view Databricks tags: From the left menu of any screen, click Governance . Under the Governance heading of the _Governance cente_r, click Tags . (Optional) Under Tags , click the funnel icon to filter tags by source type. Click Databricks to filter for tags imported from Databricks. From the left menu under Tags , select a synced tag. (Optional) Click the Linked assets tab to view linked assets for your Databricks tag. (Optional) In the top right, click the pencil icon to add a description and change the tag icon . You cannot rename tags synced from Databricks. Push tag updates to Databricks Any admin or member user in Atlan can configure reverse sync for tag updates to Databricks. You will also need to work with your Databricks administrator to grant additional permissions to push updates - you may not have access yourself. You can enable reverse sync for your imported Databricks tags in Atlan and push all tag updates for your Databricks assets back to source. Once you have enabled reverse sync, any Databricks assets with tags updated in Atlan will also be updated in Databricks. To enable reverse sync for imported Databricks tags: From the left menu of any screen, click Governance . Under the Governance heading of the _Governance cente_r, click Tags . (Optional) Under Tags , click the funnel icon to filter tags by source type. Click Databricks to filter for tags imported from Databricks. In the left menu under Tags , select a synced Databricks tag - synced tags will display the Databricks icon next to the tag name. Under Synced tags , in the upper right, turn on Enable reverse sync to synchronize tag updates from Atlan to Databricks. In the corresponding confirmation dialog, click Yes, enable it to enable reverse tag sync or click Cancel . Now when you attach Databricks tags to your Databricks assets in Atlan, these tag updates will also be pushed to Databricks! ð Enabling reverse sync will not trigger any updates in Databricks until synced tags are attached to Databricks assets in Atlan. Import Databricks tags to Atlan View Databricks tags in Atlan Push tag updates to Databricks


extract lineage and usage from Databricks | Atlan Documentation

Once you have crawled assets from Databricks , you can retrieve lineage from Unity Catalog and usage and popularity metrics from query history or system tables. This is supported for all three authentication methods : personal access token, AWS service principal, and Azure service principal. Both Atlan and Databricks strongly recommend using the system tables method to extract lineage and usage and popularity metrics from Databricks. Usage and popularity metrics can be retrieved for all Databricks users. However, your Databricks workspace must be Unity Catalog-enabled for the retrieval of lineage and usage and popularity metrics to succeed. You may also need to upgrade existing tables and views to Unity Catalog , as well as reach out to your Databricks account executive to enable lineage in Unity Catalog. (As of publishing, the feature is still in preview from Databricks on AWS and Azure.) To retrieve lineage and usage from Databricks, rev iew the order of operations and then complete the following steps. Select the extractor To select the Databricks lineage and usage extractor: In the top right of any screen, navigate to New and then click New Workflow . From the filters along the top, click Miner . From the list of packages, select Databricks Miner and click on Setup Workflow . Configure the lineage extractor Choose your lineage extraction method: In REST API , Atlan connects to your database and extracts lineage directly. In Offline , you will need to first extract lineage yourself and make it available in S3 . In System Table , Atlan connects to your database and queries system tables to extract lineage directly. REST API To configure the Databricks lineage extractor: For Connection , select the connection to extract. (To select a connection, the crawler must have already run.) Click Next to proceed. Offline extraction method Atlan supports the offline extraction method for extracting lineage from Databricks This method uses Atlan's databricks-extractor tool to extract lineage. You will need to first extract lineage yourself and make it available in S3 . To enter your S3 details: For Connection , select the connection to extract. (To select a connection, the crawler must have already run.) For Bucket name , enter the name of your S3 bucket. For Bucket prefix , enter the S3 prefix under which all the metadata files exist. These include extracted-lineage/result-0.json , extracted-query-history/result-0.json , and so on. For Bucket region , enter the name of the S3 region. When complete, at the bottom of the screen, click Next . System table To configure the Databricks lineage extractor: For Connection , select the connection to extract. (To select a connection, the crawler must have already run.) * Extraction Catalog Type : Default : Select to fetch lineage from the system catalog and access schema. Cloned_catalog : Select to fetch lineage from a cloned catalog and schema. Before proceeding, make sure the following prerequisites are met: You have already created cloned views named column_lineage and table_lineage in your schema. If not, follow the steps in Create cloned views of system tables . The atlan-user must have SELECT permissions on both views to access lineage data. Then, provide values for the following fields: Cloned Catalog Name Catalog containing the cloned views. Cloned Schema Name Schema containing the cloned views. Default : Select to fetch lineage from the system catalog and access schema. Default : Select to fetch lineage from the system catalog and access schema. Cloned_catalog : Select to fetch lineage from a cloned catalog and schema. Before proceeding, make sure the following prerequisites are met: You have already created cloned views named column_lineage and table_lineage in your schema. If not, follow the steps in Create cloned views of system tables . The atlan-user must have SELECT permissions on both views to access lineage data. Then, provide values for the following fields: Cloned Catalog Name Catalog containing the cloned views. Cloned Schema Name Schema containing the cloned views. Cloned_catalog : Select to fetch lineage from a cloned catalog and schema. Before proceeding, make sure the following prerequisites are met: You have already created cloned views named column_lineage and table_lineage in your schema. If not, follow the steps in Create cloned views of system tables . The atlan-user must have SELECT permissions on both views to access lineage data. Then, provide values for the following fields: Cloned Catalog Name Catalog containing the cloned views. Cloned Schema Name Schema containing the cloned views. For SQL Warehouse ID , enter the ID you copied from your SQL warehouse . Click Next to proceed. (Optional) Configure the usage extractor Atlan extracts usage and popularity metrics from: This feature is currently limited to queries on SQL warehouses - queries on interactive clusters are not supported. Additionally, expensive queries and compute costs for Databricks assets are currently unavailable due to limitations of the Databricks APIs . To configure the Databricks usage and popularity extractor: For Fetch Query History and Calculate Popularity , click Yes to retrieve usage and popularity metrics for your Databricks assets. For Popularity Extraction Method : Choose one of the following methods to extract usage and popularity metrics:: Click REST API to extract usage and popularity metrics from query history. Click System table to extract metrics directly from system tables: Extraction catalog type for popularity : Choose where to fetch popularity data from: Default : Uses the system catalog and query schema to fetch popularity metrics. Cloned_catalog : Select to fetch popularity from cloned views in a separate catalog and schema. Before proceeding: The query_history view must exist in the provided schema. The atlan-user must have SELECT permission on the view. Then provide: Cloned Catalog Name The catalog that contains the query_history view. Cloned Schema Name The schema that contains the query_history view. For more information, see Create cloned views of system tables . For SQL Warehouse ID , enter the ID you copied from your SQL warehouse . Click REST API to extract usage and popularity metrics from query history. Click System table to extract metrics directly from system tables: Extraction catalog type for popularity : Choose where to fetch popularity data from: Default : Uses the system catalog and query schema to fetch popularity metrics. Cloned_catalog : Select to fetch popularity from cloned views in a separate catalog and schema. Before proceeding: The query_history view must exist in the provided schema. The atlan-user must have SELECT permission on the view. Then provide: Cloned Catalog Name The catalog that contains the query_history view. Cloned Schema Name The schema that contains the query_history view. For more information, see Create cloned views of system tables . For SQL Warehouse ID , enter the ID you copied from your SQL warehouse . Extraction catalog type for popularity : Choose where to fetch popularity data from: Default : Uses the system catalog and query schema to fetch popularity metrics. Cloned_catalog : Select to fetch popularity from cloned views in a separate catalog and schema. Before proceeding: The query_history view must exist in the provided schema. The atlan-user must have SELECT permission on the view. Then provide: Cloned Catalog Name The catalog that contains the query_history view. Cloned Schema Name The schema that contains the query_history view. For more information, see Create cloned views of system tables . Extraction catalog type for popularity : Choose where to fetch popularity data from: Default : Uses the system catalog and query schema to fetch popularity metrics. Cloned_catalog : Select to fetch popularity from cloned views in a separate catalog and schema. Before proceeding: The query_history view must exist in the provided schema. The atlan-user must have SELECT permission on the view. Cloned Catalog Name The catalog that contains the query_history view. Cloned Schema Name The schema that contains the query_history view. For more information, see Create cloned views of system tables . For SQL Warehouse ID , enter the ID you copied from your SQL warehouse . For SQL Warehouse ID , enter the ID you copied from your SQL warehouse . Configure the usage extractor: For Popularity Window (days) , 30 days is the maximum limit. You can set a shorter popularity window of less than 30 days. For Start time , choose the earliest date from which to mine query history. If you're using the offline extraction method to extract query history from Databricks, skip to the next step. For Excluded Users , type the names of users to be excluded while calculating usage metrics for Databricks assets. Press enter after each name to add more names. For Popularity Window (days) , 30 days is the maximum limit. You can set a shorter popularity window of less than 30 days. For Start time , choose the earliest date from which to mine query history. If you're using the offline extraction method to extract query history from Databricks, skip to the next step. For Excluded Users , type the names of users to be excluded while calculating usage metrics for Databricks assets. Press enter after each name to add more names. If running the miner for the first time, Atlan recommends setting a start date around three days prior to the current date and then scheduling it daily to build up to two weeks of query history. Mining two weeks of query history on the first miner run may cause delays. For all subsequent runs, Atlan requires a minimum lag of 24 to 48 hours to capture all the relevant transformations that were part of a session. Learn more about the miner logic here . Run the extractor To run the Databricks lineage and popularity extractor, after completing the steps above: To check for any permissions or other configuration issues before running the crawler, click Preflight checks . This is currently only supported when using REST API and offline extraction methods. If you're using system tables, skip to step 2. You can either: To run the crawler once immediately, at the bottom of the screen, click the Run button. To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule Run button. To run the crawler once immediately, at the bottom of the screen, click the Run button. To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule Run button. Once the extractor has completed running, you will see lineage for Databricks assets! ð Select the extractor Configure the lineage extractor (Optional) Configure the usage extractor Run the extractor


Interpret usage metrics | Atlan Documentation

Atlan currently supports usage and popularity metrics for the following connectors: Amazon Redshift - tables, views, and columns. Expensive queries and compute costs for Amazon Redshift assets are currently unavailable due to limitations at source. Databricks - tables, views, and columns. Expensive queries and compute costs for Databricks assets are currently unavailable due to limitations of the Databricks APIs . Google BigQuery - tables, views, and columns Microsoft Power BI - reports and dashboards Snowflake - tables, views, and columns Powered by Atlan's enhanced query-mining capabilities, you can view popularity metrics for supported assets: The popularity score of an asset is computed using both the number of queries and the number of users who have queried that asset in the last 30 days. The popularity score of an asset helps determine its relative popularity. All assets with a popularity score are then slotted into one of four percentile groups - Least popular , Less popular , Popular , and Most popular . Popularity score is calculated using the following formula: number of distinct users * log (total number of read queries) Time period = 30 days Popularity score is calculated using the following formula: number of distinct users * log (total number of read queries) Popularity score is calculated using the following formula: Time period = 30 days Time period = 30 days The popularity indicator is displayed for all supported assets that have been queried in the last 30 days. This indicator visualizes the relative popularity of an asset on a scale of 1 to 4 blue bars - 1 being the lowest score and 4 being the highest. View popularity metrics To view popularity metrics for your assets, complete these steps. Identify popular assets Being able to identify your most relevant and trusted data assets can help you increase their adoption and drive usage within your organization. To view popularity metrics for an asset: From the left menu in Atlan, click Assets . For Connector on the Assets page, select a supported connector - for this example, we'll select Snowflake . From the Popularity sorting menu, click Most popular to view most used assets or Least popular to view least used assets. Your assets will now have a popularity indicator. To view the popularity popover for an asset, click or hover over the popularity indicator . You'll now be able to see all the relevant popularity metrics for your asset! ð View usage metrics in the asset sidebar The new Usage tab in the asset sidebar helps you view usage metadata for your assets. For example, if you'd like to appoint a data steward for your data assets, you'll be able to determine the right candidate based on the top users for that asset. You'll also be able to review popular queries or users for a particular table while checking for data compliance. To view usage details for an asset: From the left menu in Atlan, click Assets . For Connector on the Assets page, select a supported connector - for this example, we'll select Snowflake . From the Popularity sorting menu, click Most popular to view most used assets or Least popular to view least used assets. In the bottom right of any asset card, click or hover over the popularity indicator to open the popularity popover. In the popularity popover, click View usage details to view the following: For Usage , view top and recent users in the last 30 days. For Queries , view top five queries by context - Popular , Slow , and Expensive . Only read queries or SELECT statements are shown for these queries. For Compute , view the total compute cost for an asset. The compute cost is split between read and write queries, allowing you to better understand the cost breakdown for individual assets: Read queries - SELECT statements. Write queries - all non- SELECT statements, for example, UPDATE , INSERT , CREATE , and more. For Usage , view top and recent users in the last 30 days. For Queries , view top five queries by context - Popular , Slow , and Expensive . Only read queries or SELECT statements are shown for these queries. For Compute , view the total compute cost for an asset. The compute cost is split between read and write queries, allowing you to better understand the cost breakdown for individual assets: Read queries - SELECT statements. Write queries - all non- SELECT statements, for example, UPDATE , INSERT , CREATE , and more. Read queries - SELECT statements. Write queries - all non- SELECT statements, for example, UPDATE , INSERT , CREATE , and more. The usage details for the asset will now appear in the asset sidebar! ð View and sort columns by popularity For any Snowflake, Databricks, or Google BigQuery table or view sorted by popularity, you'll also be able to view and sort the columns by popularity in the asset profile. To view column assets by popularity: From the left menu in Atlan, click Assets . For Connector on the Assets page, select a supported connector - for this example, we'll select Snowflake . From the Popularity sorting menu, click Most popular to view most used assets or Least popular to view least used assets. Click any asset to open to its asset profile. In the Column preview tab of the asset profile, hover over the popularity indicator to view the popularity popover for your columns. You'll now be able to view the popularity score, number of queries and users, and timestamp for last queried for your columns! ð View queries by context Get the context you need before querying an asset to help you optimize your queries. Query popular, slow, or expensive queries from the Usage tab directly in Insights. To view and work with queries by context: From the left menu in Atlan, click Assets . For Connector on the Assets page, select a supported connector - for this example, we'll select Snowflake . From the Popularity sorting menu, click Most popular to view most used assets or Least popular to view least used assets. In the bottom right of any asset card, click or hover over the popularity indicator to open the popularity popover. In the popularity popover, click View usage details . In the Usage tab in the asset sidebar, navigate to Queries and depending on the type of query you'd like to see: Click Popular to see the top five most popular queries. Click Slow to see queries sorted by average duration and last run. Click Expensive to see the top five most expensive queries. Click Popular to see the top five most popular queries. Click Slow to see queries sorted by average duration and last run. Click Expensive to see the top five most expensive queries. Once you've selected the relevant query type, hover over a query card to: Click the expand icon to see the query details. Click the copy icon to copy the query and use it as a template for writing your own queries. Click the code icon to open the query directly in Insights and run it. Click the expand icon to see the query details. Click the copy icon to copy the query and use it as a template for writing your own queries. Click the code icon to open the query directly in Insights and run it. If you have any questions about usage and popularity metrics, head over here . View popularity metrics View queries by context


Set up Microsoft Power BI | Atlan Documentation

Depending on the authentication method you choose, you may need a combination of your Cloud Application Administrator or Application Administrator for Microsoft Entra ID, Microsoft 365 administrator for Microsoft 365, and Fabric Administrator ( formerly known as Power BI Administrator ) for Microsoft Power BI to complete these tasks -> you may not have access yourself. This guide outlines how to set up Microsoft Power BI so it can connect with Atlan for metadata extraction and lineage tracking. Before you begin Register application in Microsoft Entra ID You need your Cloud Application Administrator or Application Administrator to complete these steps > you may not have access yourself. This is required if the creation of registered applications isn't enabled for the entire organization. To register a new application in Microsoft Entra ID: Log in to the Azure portal . Click App registrations from the left menu. Click + New registration . Enter a name for your client application and click Register . Application (client) ID Directory (tenant) ID Click Certificates & secrets from the left menu. Under Client secrets , click + New client secret . Enter a description, select an expiry time, and click Add . Copy and securely store the client secret Value . Create security group in Microsoft Entra ID You need your Cloud Application Administrator or Application Administrator to complete these steps - you may not have access yourself. To create a security group for your application: Log in to the Azure portal . Click Groups under the Manage section. Click New group . Set the Group type to Security . Enter a Group name and optional description. Click No members selected . Click Select and then Create . By the end of these steps, you have registered an application with Microsoft Entra ID and created a Security Group with the appropriate member. Configure authentication options Atlan supports two authentication methods for fetching metadata from Microsoft Power BI: Service principal authentication (recommended) When using Service Principal authentication, you must decide how the connector shall access metadata to catalog assets and build lineage. There are two supported options: This option grants permissions that let the service principal to access only admin-level Power BI APIs. In this mode, Atlan extracts metadata exclusively using administrative endpoints. This option is recommended for stricter access control environments. You need your Fabric Administrator ( formerly known as Power BI Administrator ) to complete these tasks - you may not have access yourself. To configure admin API access: Log in to the Power BI admin portal . Click Tenant settings under Admin portal. Under Admin API settings : Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group under Specific security groups Click Apply Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group under Specific security groups Click Apply Add your security group under Specific security groups Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Add your security group This option grants permissions that let the service principal to access both admin and non-admin Power BI APIs. This enables Atlan to extract richer metadata and build detailed lineage across Power BI assets. You need to be at least a member of the Microsoft Power BI workspace to which you want to add the security group to complete these steps - you may not have access yourself. Make sure that you add the security group from the homepage and not the admin portal. To assign a Microsoft Power BI workspace role to the security group: Open the Microsoft Power BI homepage . Open Workspaces and select the workspace you want to access from Atlan. Click Access . In the panel: Enter the name of your security group where it says Enter email addresses Choose one of the following roles: Viewer : For workspaces without parameters Contributor : For workspaces with semantic models containing parameters or to generate lineage for measures Member : To generate lineage for dataflows Click Add . Enter the name of your security group where it says Enter email addresses Choose one of the following roles: Viewer : For workspaces without parameters Contributor : For workspaces with semantic models containing parameters or to generate lineage for measures Member : To generate lineage for dataflows Viewer : For workspaces without parameters Contributor : For workspaces with semantic models containing parameters or to generate lineage for measures Member : To generate lineage for dataflows Click Add . You need your Fabric Administrator ( formerly known as Power BI Administrator ) to complete these tasks - you may not have access yourself. To enable both admin and non-admin API access: Log in to the Power BI admin portal . Click Tenant settings under Admin portal. Under Developer settings : Expand Service principals can use Fabric APIs and set to Enabled Add your security group under Specific security groups Click Apply Expand Service principals can use Fabric APIs and set to Enabled Add your security group under Specific security groups Click Apply Add your security group under Specific security groups Under Admin API settings : Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Add your security group After making these changes, you typically need to wait 15-30 minutes for the settings to take effect across Microsoft's services. Delegated user authentication Atlan doesn't recommend using delegated user authentication as it's also not recommended by Microsoft. You need your Microsoft 365 administrator to complete these steps - you may not have access yourself. To assign the delegated user to the Fabric Administrator role: Open the Microsoft 365 admin portal . Click Users and then Active users from the left menu. Select the delegated user. Under Roles , click Manage roles . Expand Show all by category . Under Collaboration , select Fabric Administrator . Click Save changes . You need your Cloud Application Administrator or Application Administrator to complete these steps, you may not have access yourself. The following permissions are only required for delegated user authentication. If using service principal authentication, you don't need to configure any delegated permissions for a service principal it's recommended that you avoid adding these permissions. These are never used and can cause errors that may be hard to troubleshoot. To add permissions for the registered application : In your app registration, click API permissions under the Manage section. Click Add a permission . Click Delegated permissions and select: Capacity.Read.All Dataflow.Read.All Dataset.Read.All Report.Read.All Tenant.Read.All Workspace.Read.All Click Grant Admin consent (If you only see the Add permissions button, you aren't an administrator). You need your Fabric Administrator ( formerly known as Power BI Administrator ) to complete these tasks, you may not have access yourself. To enable the Microsoft Power BI admin API: Log in to the Power BI admin portal . Click Tenant settings under Admin portal. Under Admin API settings : Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply . Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply . Add your security group Click Apply . Before you begin Configure authentication options


Atlan architecture | Atlan Documentation

Atlan is a cloud-first solution. Single-tenant SaaS is the recommended deployment model. Atlan currently supports hosting tenants on the following cloud platforms: Amazon Web Services (AWS) Microsoft Azure Google Cloud Platform (GCP) The components of Atlan are isolated, across both compute and data. For more details, see How are resources isolated? Platform components Kong is an API gateway. It handles rate limiting and token verification on all incoming API requests. Apache Keycloak is an identity and access management component. It manages everything to do with users, login, SSO and so on. Heracles is Atlan's API service. It houses the business logic used by the frontend and APIs to interact with other platform components. PostgreSQL is a SQL database. Many services on the platform use it for storage. HashiCorp Vault is a secret manager. It stores sensitive credentials provided by the user. Apache Ranger is the policy engine. It provides fine-grained access control over data in the metastore. Argo Workflows is a workflow orchestrator for k8s. It runs and manages long-running jobs in a container and k8s-native fashion. Admission Controller is a k8s admission controller. It performs certain actions when Argo Workflows are updated such as workflow alerts. Apache Zookeeper manages consensus and coordination for the metastore services. Apache Cassandra is an object-oriented database used to store the metastore's data. Apache Kafka is an event stream. It enables event-driven use cases across the platform. Heka is Atlan's SQL component. It parses, rewrites and optimizes SQL queries and is powered by Apache Calcite . Redis is a cache layer used by Heracles. Platform management components Velero performs cluster backups. Fluent Bit is a logging and metrics processor. It parses and pushes logs from pods to various destinations. Central components Zenduty is used for incident response. Alerts are sent when something goes wrong in one of the clusters. Argo CD is used for continuous deployment. Changes in git repositories lead to upgrades in the clusters. Github Actions update the Docker container images as part of the development process. Sendgrid is used to send emails. The frontend is a Vue.js web application that's hosted on S3 and delivered via Amazon CloudFront content delivery network (CDN) service. Alertmanager sends alerts generated by metrics stored in Prometheus. Grafana provides observability dashboards. VictoriaMetrics is a fast, cost-effective, and scalable monitoring solution and time series database. It processes high volumes of data and enables long-term storing. Atlan marketplace (not pictured) The marketplace offers packages (workflows) that perform long-running tasks on the Atlan platform. The ecosystem enables the creation of metadata and lineage connectors. See security.atlan.com for the latest policies and standards, reports and certifications, architecture, diagrams and more. Platform management components Atlan marketplace (not pictured)


How are resources isolated? | Atlan Documentation

Each Atlan customer has their own isolated set of nodes within Kubernetes. The underlying Kubernetes control plane and networking layer (coredns) are shared between tenants. To achieve logical isolation, Loft s virtual clusters are implemented. The underlying Kubernetes control plane and networking layer (coredns) are shared between tenants. To achieve logical isolation, Loft s virtual clusters are implemented. The compute resources (nodes, nodegroups) and storage are physically isolated between tenants. The compute resources (nodes, nodegroups) and storage are physically isolated between tenants. Only Atlan's cloud team is able to manage the AWS , Azure , and GCP resources across these levels of isolation. Only Atlan's cloud team is able to manage the AWS , Azure , and GCP resources across these levels of isolation. See security.atlan.com for the latest policies and standards, reports and certifications, architecture, diagrams and more.