Customer support | Atlan Documentation

One of Atlan's core values is to help you and your team do your life's best work. ð That's why Atlan wants to make it as easy as possible for you to keep driving your work forward with data. Atlan's customer support is a combination of several teams in Atlan: Product support personnel Cloud support personnel DevOps/engineering support personnel Vast repository of self-service resources Service-level commitment Atlan's Technical Support team provides support globally with high response commitment levels. This includes 24/7 SRE support for critical (P0) issues. Customers get a service-level commitment, including the following: 99.5% uptime for Atlan Dedicated support center, available from within the Atlan product Commitments for aggressive response times for business critical issues Designated Customer Success Manager to assist with escalations Ways to contact support ï¸ Email support at a dedicated customer support email account ( [email protected] ) ð ¨ ð » In-product support widget to log tickets and a help desk portal to log and track tickets. You can sign up to track support tickets on the help desk portal. You must use your organizational email address as the username and create a password. ð Submit a support request via the online form. To track your support tickets: Navigate to https://atlan.zendesk.com and log into the help desk portal with your credentials or via SSO. From the top right, click your avatar, and then from the dropdown, click My activities . On the My activities page, you can do the following: My requests and Requests I'm CC'd on - view and edit the support tickets you either created or were copied on, respectively. Organization requests - to access all other support tickets for your organization, please reach out to your customer success manager. Atlan provides you with read access to all the support tickets for your organization. To be able to comment on or close them, you must be CC'd on all tickets. My requests and Requests I'm CC'd on - view and edit the support tickets you either created or were copied on, respectively. Organization requests - to access all other support tickets for your organization, please reach out to your customer success manager. Atlan provides you with read access to all the support tickets for your organization. To be able to comment on or close them, you must be CC'd on all tickets. Hours of operation 24x7 availability for all requests and issues Severity levels The Atlan Technical Support team determines the severity of an issue. The customer's position is considered, and these guidelines are followed to determine priority. Below are the response time SLAs: Escalation procedure If the business impact of a support request changes or a ticket isn't being handled according to your expectations, you may escalate the ticket. Please first speak with the Technical Support representative assigned to the ticket to confirm that the business impact and urgency are understood. You may further escalate by contacting: 1st level of escalation : Technical Support Engineer 2nd level of escalation : Director, Support 3rd level of escalation : Head of Customer Experience Ways to contact support Hours of operation


Developing with Atlan - Developer

Overall site map If you are new to Atlan, or to developing with Atlan, start with one of the following two options. These will set you up to develop with Atlan, step-by-step. Atlan University Self-paced, video-based walkthrough of the essentials of Atlan as a platform. Atlan Platform Essentials certification Self-paced, video-based walkthrough of the essentials of Atlan as a platform. Atlan Platform Essentials certification Introductory walkthrough Documentation-based walkthrough, including step-by-step examples. Start the walkthrough Documentation-based walkthrough, including step-by-step examples. Start the walkthrough Common tasks Common operations on assets, that are available across all assets. Discover actions Common operations on assets, that are available across all assets. Asset-specific Operations that are specific to certain assets. Focus on a specific kind of asset Operations that are specific to certain assets. Focus on a specific kind of asset Governance structures Operations dealing with governance structures, rather than assets. Manage governance structures Operations dealing with governance structures, rather than assets. Manage governance structures Samples Real code samples our customers use to solve particular use cases. Review live samples Real code samples our customers use to solve particular use cases. Review live samples Events Delve deep into the details of the events Atlan triggers. Learn more about events Delve deep into the details of the events Atlan triggers. Learn more about events


Set up Snowflake | Atlan Documentation

You need your Snowflake administrator to run these commands - you may not have access yourself. Create user and role in Snowflake Create a role and user in Snowflake using the following commands: Create role Create a role in Snowflake using the following commands: Replace with the default warehouse to use when running the Snowflake crawler . Atlan requires the following privileges to: OPERATE enables Atlan to start the virtual warehouse to fetch metadata if the warehouse has stopped. USAGE enables Atlan to show or list metadata from Snowflake. This in turn enables the Snowflake crawler to run the SHOW query. Create a user Create a separate user to integrate into Atlan, using one of the following 3 options: See Snowflake's official guide for details on generating an RSA key-pair . To create a user with a key-pair, replace the value for rsa_public_key with the public key and run the following: Learn more about the SERVICE type property in Snowflake documentation . Atlan only supports encrypted private keys with a non-empty passphrase - generally recommended as more secure. An empty passphrase results in workflow failures. To generate an encrypted private key, omit the -nocrypt option. Refer to Snowflake documentation to learn more. Snowflake recommends transitioning away from basic authentication using username and password. Change to key-pair authentication for enhanced security. For any existing Snowflake workflows, you can modify the crawler configuration to update the authentication method. To create a user with a password, replace and run the following: Learn more about the LEGACY_SERVICE type property in Snowflake documentation . This method is currently only available if Okta is your IdP (Snowflake supports) authenticating natively through Okta : Create a user in your identity provider (IdP) and use federated authentication in Snowflake . The password for this user must be maintained solely in the IdP and multi-factor authentication (MFA) must be disabled. Grant role to user To grant the atlan_user_role to the new user: Configure OAuth (client credentials flow) with Microsoft Entra ID To configure OAuth authentication using Microsoft Entra ID (formerly Azure AD) with the client credentials flow: Follow Snowflake's documentation to: Register a new application in Microsoft Entra ID Collect the client ID , tenant ID , and client secret Add the required API permissions Follow Snowflake's documentation to: Register a new application in Microsoft Entra ID Collect the client ID , tenant ID , and client secret Add the required API permissions In Snowflake, create a security integration using the following: CREATE SECURITY INTEGRATION external_oauth_azure_ad TYPE = external_oauth ENABLED = true EXTERNAL_OAUTH_TYPE = azure EXTERNAL_OAUTH_ISSUER = '\ ' EXTERNAL_OAUTH_JWS_KEYS_URL = '\ ' EXTERNAL_OAUTH_AUDIENCE_LIST = ( '\ ' ) EXTERNAL_OAUTH_TOKEN_USER_MAPPING_CLAIM = 'sub' EXTERNAL_OAUTH_SNOWFLAKE_USER_MAPPING_ATTRIBUTE = 'login_name' ; Replace the placeholders with actual values from your Azure AD app: Your tenant's OAuth 2.0 issuer URL Azure JWKs URI Application ID URI of the Azure app In Snowflake, create a security integration using the following: Replace the placeholders with actual values from your Azure AD app: Your tenant's OAuth 2.0 issuer URL Azure JWKs URI Application ID URI of the Azure app Create a Snowflake user with a login name that exactly matches the Azure AD client object ID: CREATE USER oauth_svc_user WITH LOGIN_NAME = '\ ' -- Use Azure client OBJECT ID DEFAULT_ROLE = \ DEFAULT_WAREHOUSE = \ ; Create a Snowflake user with a login name that exactly matches the Azure AD client object ID: Grant the configured role to this user: GRANT ROLE \ TO USER oauth_svc_user ; Grant the configured role to this user: Choose metadata fetching method Atlan supports two methods for fetching metadata from Snowflake - account usage and information schema. You should choose one of these two methods to set up Snowflake: Grant permissions for account usage method If you want to set warehouse timeouts when using this method, set a large value initially for the workflow to succeed. Once the workflow has succeeded, adjust the value to be twice the extraction time. This method uses the views in SNOWFLAKE.ACCOUNT_USAGE (or a copied version of this schema) to fetch the metadata from Snowflake into Atlan. You can be more granular with permissions using this method, but there are limitations with this approach . To crawl assets, generate lineage, and import tags If you also want to be able to preview and query the data, you can use the preview and query existing assets permissions instead. Snowflake stores all tag objects in the ACCOUNT_USAGE schema. If you're using the account usage method to crawl metadata in Atlan or you have configured the Snowflake miner , you need to grant the same permissions to import tags as required for crawling Snowflake assets. Note that object tagging in Snowflake currently requires Enterprise Edition or higher . To use the default SNOWFLAKE database and ACCOUNT_USAGE schema and also mine Snowflake's query history (for lineage), grant these permissions: The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: Replace with the copied Snowflake database name. Replace with the copied Snowflake ACCOUNT_USAGE schema name. The grants for the copied version can't be used on the original SNOWFLAKE database. This is because Snowflake produces an error that granular grants can't be given to imported databases. When using a cloned or copied version, verify that the table or view definition remains unchanged as in your SNOWFLAKE database. If the format is different. For example, a column is missing and it no longer qualifies as a clone. To crawl streams To crawl streams, provide the following permissions: To crawl current streams: Replace with the Snowflake database name. Replace with the Snowflake database name. To crawl future streams: To crawl future streams: Replace with the Snowflake database name. (Optional) To preview and query existing assets To query and preview data within assets that already exist in Snowflake, add these permissions: Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you wish to preview and query in Atlan.) (Optional) To preview and query future assets To query and preview data within assets that may be created in the future in Snowflake, add these permissions. Again, if you want to also be able to preview and query the data for future assets, you can add the preview and query future assets permissions instead. Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you want to preview and query in Atlan.) Verify that all the assets you'd like to crawl are present in these grants by checking the grants on the user role defined for the crawler. Grant permissions for information schema method This method uses views in the INFORMATION_SCHEMA schema in Snowflake databases to fetch metadata. You still need to grant specific permissions to enable Atlan to crawl metadata, preview data, and query data with this method. To crawl existing assets Grant these permissions to crawl assets that already exist in Snowflake. If you also want to be able to preview and query the data, you can use the preview and query existing assets permissions instead. Grant permissions to crawl existing assets: Replace with the database you want to be available in Atlan. (Repeat the statements for every database you wish to integrate into Atlan.) Grant permissions to crawl functions: Replace with the database you want to be available in Atlan. (Repeat the statements for every database you wish to integrate into Atlan.) For secure user-defined functions (UDFs), grant OWNERSHIP permissions to retrieve metadata: Replace the placeholders with the appropriate values: : The name of the schema that contains the user-defined function (UDF). : The name of the secure UDF that requires ownership permissions. : The role that gets assigned ownership of the secure UDF. The statements given on this page apply to all schemas, tables, and views in a database in Snowflake. If you want to limit access to only certain objects, you can instead specify the exact objects individually as well. To crawl future assets To crawl assets that may be created in the future in Snowflake, add these permissions. Again, if you want to also be able to preview and query the data for future assets, you can add the preview and query future assets permissions instead. To grant permissions at a database level: Replace with the database you want to crawl in Atlan. (Repeat the statements for every database you want to integrate into Atlan.) For any future grants defined at a schema level to a different role, the schema-level grants take precedence over the database-level grants and any future grants defined at a database level to the Atlan role get ignored. To learn more, refer to Snowflake documentation . To grant permissions at a schema level: Replace with the database and with the schema you want to crawl in Atlan. (Repeat the statements for every database and schema you want to integrate into Atlan.) To mine query history for lineage To also mine Snowflake's query history (for lineage), add these permissions. You can use either option: To mine query history direct from Snowflake's internal tables: To mine query history from a cloned or copied set of tables, where you can also remove any sensitive data: Replace with the name of the cloned database, and with the name of the cloned schema containing account usage details. When using a cloned or copied version, verify that the table or view definition remains unchanged as in your SNOWFLAKE database. If the format is different. For example, a column is missing and it no longer qualifies as a clone. (Optional) To preview and query existing assets To query and preview data within assets that already exist in Snowflake, add these permissions: Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you wish to preview and query in Atlan.) (Optional) To preview and query future assets To query and preview data within assets that may be created in the future in Snowflake, add these permissions. Again, if you want to also be able to preview and query the data for future assets, you can add the preview and query future assets permissions instead. Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you want to preview and query in Atlan.) For any future grants defined at a schema level to a different role, the schema-level grants take precedence over the database-level grants and any future grants defined at a database level to the Atlan role get ignored. To learn more, refer to Snowflake documentation . To grant permissions at a schema level: Replace with the database and with the schema you want to be able to preview and query in Atlan. (Repeat the statements for every database and schema you want to preview and query in Atlan.) Verify that all the assets you'd like to crawl are present in these grants by checking the grants on the user role defined for the crawler. (Optional) To import Snowflake tags Snowflake stores all tag objects in the ACCOUNT_USAGE schema. Note that object tagging in Snowflake currently requires Enterprise Edition or higher . To import tags from Snowflake , grant these permissions: To use the default SNOWFLAKE database and ACCOUNT_USAGE schema and also mine Snowflake's query history (for lineage), grant these permissions: The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: Replace with the copied Snowflake database name. Replace with the copied Snowflake ACCOUNT_USAGE schema name. The grants for the copied version can't be used on the original SNOWFLAKE database. This is because Snowflake produces an error that granular grants can't be given to imported databases. (Optional) To push updated tags to Snowflake To push tags updated for assets in Atlan to Snowflake , grant these permissions: You can learn more about tag privileges from Snowflake documentation . (Optional) To crawl dynamic tables Atlan currently supports fetching metadata for dynamic tables using the MONITOR privilege. Refer to Snowflake documentation to learn more. To crawl existing dynamic tables from Snowflake: Grant permissions at a database level: Grant permissions at a schema level: To crawl future dynamic tables from Snowflake: Grant permissions at a database level: Grant permissions at a schema level: Replace with the database and with the schema you want to crawl in Atlan. (Repeat the statements for every database and schema you want to integrate into Atlan.) (Optional) To crawl Iceberg tables Atlan currently supports fetching metadata for Iceberg tables only for the information schema extraction method. To crawl Iceberg tables from Snowflake, grant the following permissions: To crawl existing Iceberg tables in Snowflake: To crawl future Iceberg tables in Snowflake: To crawl Iceberg catalog metadata for Iceberg tables in Snowflake: You must first grant permissions to crawl existing Iceberg tables for this permission to work on catalogs. You must also grant permissions to all the catalogs you want to crawl in Atlan individually. (Optional) To crawl Snowflake stages Atlan supports crawling metadata for Snowflake stages using the USAGE and READ privileges. For more information, see the Snowflake documentation for INFORMATION_SCHEMA.STAGES. To crawl stages from Snowflake: Grant USAGE and READ privileges on all existing stages at the database level: Replace with the name of your Snowflake database Replace with the name of your Snowflake database Replace with the role you've granted Atlan to use for crawling. Replace with the role you've granted Atlan to use for crawling. Grant USAGE and READ privileges on all future stages at the database level: Grant USAGE and READ privileges on all future stages at the database level: Replace with the name of your Snowflake database Replace with the role you've granted Atlan to use for crawling. Allowlist the Atlan IP If you are using the IP allowlist in your Snowflake instance, you must add the Atlan IP to the allowlist. Please raise a support ticket from within Atlan, or submit a request . (If you aren't using the IP allowlist in your Snowflake instance, you can skip this step.) Create user and role in Snowflake Choose metadata fetching method Grant permissions for account usage method Grant permissions for information schema method Allowlist the Atlan IP


Set up Snowflake | Atlan Documentation


Crawl Snowflake | Atlan Documentation

Once you have configured the Snowflake user permissions , you can establish a connection between Atlan and Snowflake. (If you are also using AWS PrivateLink or Azure Private Link for Snowflake, you will need to set that up first, too.) To crawl metadata from Snowflake, review the order of operations and then complete the following steps. Select the source To select Snowflake as your source: In the top right of any screen, navigate to New and then click New Workflow . From the list of packages, select Snowflake Assets and click on Setup Workflow . Provide credentials Choose your extraction method: In Direct extraction, Atlan connects to your database and crawls metadata directly. In Offline extraction, you will need to first extract metadata yourself and make it available in S3 . This is currently only supported when using the information schema extraction method to fetch metadata with basic authentication . In Agent extraction, Atlan's secure agent executes metadata extraction within the organization's environment. Direct extraction method To enter your Snowflake credentials: For Account Identifiers (Host) , enter the hostname, AWS PrivateLink endpoint , or Azure Private Link endpoint for your Snowflake instance. For Authentication , choose the method you configured when setting up the Snowflake user : For Basic authentication, enter the Username and Password you configured in either Snowflake or the identity provider. info ð ª Did you know? Snowflake recommends transitioning away from basic authentication using username and password. Change to key-pair authentication for enhanced security. For any existing Snowflake workflows, you can modify the crawler configuration to update the authentication method. For Keypair authentication, enter the Username , Encrypted Private Key , and Private Key Password you configured. Atlan only supports encrypted private keys with a non-empty passphrase - generally recommended as more secure. An empty passphrase will result in workflow failures. To generate an encrypted private key, refer to Snowflake documentation . For Okta SSO authentication, enter the Username , Password , and Authenticator you configured. The Authenticator will be the Okta URL endpoint of your Okta account , typically in the form of https:// .okta.com . For Basic authentication, enter the Username and Password you configured in either Snowflake or the identity provider. info ð ª Did you know? Snowflake recommends transitioning away from basic authentication using username and password. Change to key-pair authentication for enhanced security. For any existing Snowflake workflows, you can modify the crawler configuration to update the authentication method. For Basic authentication, enter the Username and Password you configured in either Snowflake or the identity provider. ð ª Did you know? Snowflake recommends transitioning away from basic authentication using username and password. Change to key-pair authentication for enhanced security. For any existing Snowflake workflows, you can modify the crawler configuration to update the authentication method. For Keypair authentication, enter the Username , Encrypted Private Key , and Private Key Password you configured. Atlan only supports encrypted private keys with a non-empty passphrase - generally recommended as more secure. An empty passphrase will result in workflow failures. To generate an encrypted private key, refer to Snowflake documentation . For Keypair authentication, enter the Username , Encrypted Private Key , and Private Key Password you configured. Atlan only supports encrypted private keys with a non-empty passphrase - generally recommended as more secure. An empty passphrase will result in workflow failures. To generate an encrypted private key, refer to Snowflake documentation . For Okta SSO authentication, enter the Username , Password , and Authenticator you configured. The Authenticator will be the Okta URL endpoint of your Okta account , typically in the form of https:// .okta.com . For Okta SSO authentication, enter the Username , Password , and Authenticator you configured. The Authenticator will be the Okta URL endpoint of your Okta account , typically in the form of https:// .okta.com . For Role , select the Snowflake role through which the crawler should run. For Warehouse , select the Snowflake warehouse in which the crawler should run. Click Test Authentication to confirm connectivity to Snowflake using these details. Once successful, at the bottom of the screen, click Next . Offline extraction method Atlan supports the offline extraction method for fetching metadata from Snowflake. This method uses Atlan's metadata-extractor tool to fetch metadata. You will need to first extract the metadata yourself and then make it available in S3 . To enter your S3 details: For Bucket name , enter the name of your S3 bucket. If you are reusing Atlan's S3 bucket, you can leave this blank. For Bucket prefix , enter the S3 prefix under which all the metadata files exist. These include databases.json , columns- .json , and so on. For Bucket region , enter the name of the S3 region. When complete, at the bottom of the screen, click Next . Configure the connection To complete the Snowflake connection configuration: Provide a Connection Name that represents your source environment. For example, you might use values like production , development , gold , or analytics . Provide a Connection Name that represents your source environment. For example, you might use values like production , development , gold , or analytics . (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins . danger If you do not specify any user or group, nobody will be able to manage the connection - not even admins. (Optional) To change the users able to manage this connection, change the users or groups listed under Connection Admins . If you do not specify any user or group, nobody will be able to manage the connection - not even admins. (Optional) To prevent users from querying any Snowflake data, change Allow SQL Query to No . (Optional) To prevent users from querying any Snowflake data, change Allow SQL Query to No . (Optional) To prevent users from previewing any Snowflake data, change Allow Data Preview to No . (Optional) To prevent users from previewing any Snowflake data, change Allow Data Preview to No . At the bottom of the screen, click Next to proceed. At the bottom of the screen, click Next to proceed. Agent extraction method Atlan supports using a Secure Agent for fetching metadata from Snowflake. To use a Secure Agent, follow these steps: Select the Agent tab. Configure the Snowflake data source by adding the secret keys for your secret store. For details on the required fields, refer to the Direct extraction section. Complete the Secure Agent configuration by following the instructions in the How to configure Secure Agent for workflow execution guide. Click Next after completing the configuration. Configure the crawler When modifying an existing Snowflake connection, switching to a different extraction method will delete and recreate all assets in the existing connection. If you'd like to change the extraction method, contact Atlan support for assistance. Before running the Snowflake crawler, you can further configure it. You must select the Extraction method you configured when you set up Snowflake : For Information Schema method , keep the default selection. Change to Account Usage method and specify the following: Database Name of the copied Snowflake database Schema Name of the copied ACCOUNT_USAGE schema Incremental extraction Public preview - Toggle incremental extraction for faster and more efficient metadata extraction. Database Name of the copied Snowflake database Schema Name of the copied ACCOUNT_USAGE schema Incremental extraction Public preview - Toggle incremental extraction for faster and more efficient metadata extraction. You can override the defaults for any of the remaining options: For Asset selection , select a filtering option: To select the assets you want to include in crawling, click Include by hierarchy and filter for assets down to the database or schema level. (This will default to all assets, if none are specified.) To have the crawler include Databases , Schemas , or Tables & Views based on a naming convention, click Include by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_DB.* for Databases will include all the matching databases and their child assets. To select the assets you want to exclude from crawling, click Exclude by hierarchy and filter for assets down to the database or schema level. (This will default to no assets, if none are specified.) To have the crawler ignore Databases , Schemas , or Tables & Views based on a naming convention, click Exclude by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_TABLES.* for Tables & Views will exclude all the matching tables and views. Click + to add more filters. If you add multiple filters, assets will be crawled based on matching all the filtering conditions you have set. For Asset selection , select a filtering option: To select the assets you want to include in crawling, click Include by hierarchy and filter for assets down to the database or schema level. (This will default to all assets, if none are specified.) To have the crawler include Databases , Schemas , or Tables & Views based on a naming convention, click Include by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_DB.* for Databases will include all the matching databases and their child assets. To select the assets you want to exclude from crawling, click Exclude by hierarchy and filter for assets down to the database or schema level. (This will default to no assets, if none are specified.) To have the crawler ignore Databases , Schemas , or Tables & Views based on a naming convention, click Exclude by regex and specify a regular expression - for example, specifying ATLAN_EXAMPLE_TABLES.* for Tables & Views will exclude all the matching tables and views. Click + to add more filters. If you add multiple filters, assets will be crawled based on matching all the filtering conditions you have set. To exclude lineage for views in Snowflake, change View Definition Lineage to No . To exclude lineage for views in Snowflake, change View Definition Lineage to No . To import tags from Snowflake to Atlan , change Import Tags to Yes . Note the following: If using the Account Usage extraction method, grant the same permissions as required for crawling Snowflake assets to import tags and push updated tags to Snowflake. If using the Information Schema extraction method, note that Snowflake stores all tag objects in the ACCOUNT_USAGE schema. You will need to grant permissions on the account usage schema instead to import tags from Snowflake. danger Object tagging in Snowflake currently requires Enterprise Edition or higher . If your organization does not have Enterprise Edition or higher and you try to import Snowflake tags to Atlan, the Snowflake connection will fail with an error - unable to retrieve tags. To import tags from Snowflake to Atlan , change Import Tags to Yes . Note the following: If using the Account Usage extraction method, grant the same permissions as required for crawling Snowflake assets to import tags and push updated tags to Snowflake. If using the Information Schema extraction method, note that Snowflake stores all tag objects in the ACCOUNT_USAGE schema. You will need to grant permissions on the account usage schema instead to import tags from Snowflake. Object tagging in Snowflake currently requires Enterprise Edition or higher . If your organization does not have Enterprise Edition or higher and you try to import Snowflake tags to Atlan, the Snowflake connection will fail with an error - unable to retrieve tags. For Control Config , keep Default for the default configuration or click Custom to further configure the crawler: If you have received a custom crawler configuration from Atlan support, for Custom Config , enter the value provided. You can also: Enter {"ignore-all-case": true} to enable crawling assets with case-sensitive identifiers. For Enable Source Level Filtering , click True to enable schema-level filtering at source or keep False to disable it. For Use JDBC Internal Methods , click True to enable JDBC internal methods for data extraction or click False to disable it. For Exclude tables with empty data , change to Yes to exclude any tables and corresponding columns without any data. For Exclude views , change to Yes to exclude all views from crawling. For Control Config , keep Default for the default configuration or click Custom to further configure the crawler: If you have received a custom crawler configuration from Atlan support, for Custom Config , enter the value provided. You can also: Enter {"ignore-all-case": true} to enable crawling assets with case-sensitive identifiers. Enter {"ignore-all-case": true} to enable crawling assets with case-sensitive identifiers. For Enable Source Level Filtering , click True to enable schema-level filtering at source or keep False to disable it. For Use JDBC Internal Methods , click True to enable JDBC internal methods for data extraction or click False to disable it. For Exclude tables with empty data , change to Yes to exclude any tables and corresponding columns without any data. For Exclude views , change to Yes to exclude all views from crawling. If an asset appears in both the include and exclude filters, the exclude filter takes precedence. Run the crawler To run the Snowflake crawler, after completing the steps above: To check for any permissions or other configuration issues before running the crawler, click Preflight checks . You can either: To run the crawler once immediately, at the bottom of the screen, click the Run button. To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule Run button. To run the crawler once immediately, at the bottom of the screen, click the Run button. To schedule the crawler to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule Run button. Once the crawler has completed running, you will see the assets in Atlan's asset page! ð Note that the Atlan crawler will currently skip any unsupported data types to ensure a successful workflow run. Select the source Configure the connection Configure the crawler Run the crawler


Manage connectivity | Atlan Documentation

Once you've scheduled or run a workflow you can modify its configuration at any time. The configuration that can be modified may vary by workflow but the general steps remain consistent. Modify connectivity To modify the configuration of an existing workflow, complete the following steps. On the left of any screen, navigate to Workflow . Under Monitor select an existing workflow tile. (You may need to expand the run history or filter first.) From the Workflow Run History table, click on the previous run of the workflow you want to modify. In the upper left of the screen, change to the Config tab. Modify the parts of the workflow configuration you require: Under Credential , use the Edit Credentials button to change the credentials for the source. danger If you're updating the connection credentials, you may also need to update the metadata filters before running the updated workflow. Atlan currently does not detect changes to your connection settings and update the metadata filters automatically. Under Connection settings , use the Edit button to change the connection details: Modify whether or not querying or data previews are allowed for the source. Modify the query row limit to enable exporting large query results via email . Modify the query timeout limit - expandable up to 60 minutes. Under Connection Admins , click the pencil icon to add or remove connection admins. danger If you do not specify any user or group, nobody will be able to manage the connection - not even admins. Under Metadata , use the selectors to modify which metadata to include and exclude. To check for any permissions or other configuration issues before running the workflow, click Preflight checks . Under Credential , use the Edit Credentials button to change the credentials for the source. danger If you're updating the connection credentials, you may also need to update the metadata filters before running the updated workflow. Atlan currently does not detect changes to your connection settings and update the metadata filters automatically. Under Credential , use the Edit Credentials button to change the credentials for the source. If you're updating the connection credentials, you may also need to update the metadata filters before running the updated workflow. Atlan currently does not detect changes to your connection settings and update the metadata filters automatically. Under Connection settings , use the Edit button to change the connection details: Modify whether or not querying or data previews are allowed for the source. Modify the query row limit to enable exporting large query results via email . Modify the query timeout limit - expandable up to 60 minutes. Under Connection settings , use the Edit button to change the connection details: Modify whether or not querying or data previews are allowed for the source. Modify the query row limit to enable exporting large query results via email . Modify the query timeout limit - expandable up to 60 minutes. Under Connection Admins , click the pencil icon to add or remove connection admins. danger If you do not specify any user or group, nobody will be able to manage the connection - not even admins. Under Connection Admins , click the pencil icon to add or remove connection admins. If you do not specify any user or group, nobody will be able to manage the connection - not even admins. Under Metadata , use the selectors to modify which metadata to include and exclude. Under Metadata , use the selectors to modify which metadata to include and exclude. To check for any permissions or other configuration issues before running the workflow, click Preflight checks . To check for any permissions or other configuration issues before running the workflow, click Preflight checks . Once you've made your updates, click the Update button to save the changes. You can optionally run the workflow with the new configuration immediately. You will need to confirm your changes by clicking the Yes button. Note that some workflow changes may take a few minutes to come into effect. You can optionally run the workflow with the new configuration immediately. You will need to confirm your changes by clicking the Yes button. Note that some workflow changes may take a few minutes to come into effect. That's it - next time you run the workflow, or it runs on its schedule, it will use your changes! ð If you modify the Metadata portion, any previously crawled metadata that is now excluded will be archived on the next workflow run.


Troubleshooting Snowflake connectivity | Atlan Documentation

Missing warehouse grants The user doesn t have USAGE and OPERATE grants on a warehouse. Grant warehouse access to the role : GRANT OPERATE , USAGE ON WAREHOUSE " " TO ROLE atlan_user_role ; Grant warehouse access to the role : Then, ensure that you grant the role to the new user : GRANT ROLE atlan_user_role TO USER atlan_user ; Then, ensure that you grant the role to the new user : Missing authorized access to SNOWFLAKE.ACCOUNT_USAGE schema The user doesn t have authorized access to the SNOWFLAKE.ACCOUNT_USAGE database Reach out to your account admin to grant imported privileges on the Snowflake database to the role: USE ROLE ACCOUNTADMIN ; GRANT IMPORTED PRIVILEGES ON DATABASE SNOWFLAKE TO ROLE atlan_user_role ; Reach out to your account admin to grant imported privileges on the Snowflake database to the role: If using a copied database , you'll need to grant the following permissions: GRANT USAGE ON DATABASE " " TO ROLE atlan_user_role ; GRANT USAGE ON SCHEMA " " IN DATABASE " " TO ROLE atlan_user_role ; GRANT REFERENCES ON ALL VIEWS IN DATABASE " " TO ROLE atlan_user_role ; If using a copied database , you'll need to grant the following permissions: Missing usage grants on databases and/or schemas The user doesn't have usage grants to the databases ` $missingDatabases ` and schemas ` $missingSchemas Grant missing permissions listed here for information schema extraction method. Atlan IP not allowlisted Atlan's current location or network isn't recognized by Snowflake's security settings. This can happen if Atlan's IP address isn't on the list of allowed addresses in Snowflake's network policies. If you are using the IP allowlist in your Snowflake instance, you must add the Atlan IP to the allowlist . Contact Atlan support to obtain Atlan's IP addresses. The username or the password provided to connect to the Snowflake account is incorrect. Sign into the Snowflake account for the specified host and verify that the username and password are correct. You can also create a new user, if required, by following the steps here . Missing or unauthorized role The role specified in your connection configuration doesn't exist in Snowflake or your user account doesn't have grant to use this role. If the role does not exist or is missing the required grants, create a role and then grant the role to the user . User account locked The user account you're using to connect to Snowflake has been locked temporarily because of multiple incorrect login attempts. Wait for the user account to unlock or create a different user account to continue. Missing or unauthorized warehouse The warehouse specified in your connection configuration doesn't exist in Snowflake or your user account doesn't have grant to use this warehouse. Ensure that the warehouse name is configured correctly. Ensure that the warehouse name is configured correctly. Update the warehouse name in the configuration if your account is using a different warehouse. Create a role and then grant the role to the user for the updated warehouse. Update the warehouse name in the configuration if your account is using a different warehouse. Create a role and then grant the role to the user for the updated warehouse. Missing access to non-system databases or schemas The configured user doesn't have usage grants to any database or schema. or The configured user doesn't have usage grants to any non-system database or schema. This pertains to the information schema method of fetching metadata. Ensure that the user has authorized access to the databases and schemas to be crawled. Grant the requisite permissions as outlined here . Check the grants on the role attached to the user defined for the crawler. Ensure the missing database or schema is present in these grants. SHOW GRANTS TO ROLE atlan_user_role ; Check the grants on the role attached to the user defined for the crawler. Ensure the missing database or schema is present in these grants. When using incremental extraction, consider running a one-time full extraction to capture any newly introduced metadata. When using incremental extraction, consider running a one-time full extraction to capture any newly introduced metadata. Make sure the role attached to the user defined for the crawler has grants for future tables and views being created in the database: GRANT USAGE ON FUTURE SCHEMAS IN DATABASE " " TO ROLE atlan_user_role ; GRANT REFERENCES ON FUTURE TABLES IN DATABASE " " TO ROLE atlan_user_role ; GRANT REFERENCES ON FUTURE VIEWS IN DATABASE " " TO ROLE atlan_user_role ; GRANT REFERENCES ON FUTURE EXTERNAL TABLES IN DATABASE " " TO ROLE atlan_user_role ; Make sure the role attached to the user defined for the crawler has grants for future tables and views being created in the database: Make sure you run the below commands as well so that new tables and views you've created in-between are also visible to the user: GRANT USAGE ON ALL SCHEMAS IN DATABASE " " TO role atlan_user_role ; GRANT REFERENCES ON ALL TABLES IN DATABASE " " TO role atlan_user_role ; GRANT REFERENCES ON ALL EXTERNAL TABLES IN DATABASE " " TO atlan_user_role ; GRANT REFERENCES ON ALL VIEWS IN DATABASE " " TO role atlan_user_role ; Make sure you run the below commands as well so that new tables and views you've created in-between are also visible to the user: The query miner only mines query history for up to the previous two weeks. The miner will not mine any queries that ran before that time window. If the queries that created your assets ran before that time window, lineage for those assets will not be present. To mine more than the previous two weeks of query history, either use S3-based query mining or contact Atlan support . Note that Snowflake itself only retains query history for so long as well, though. Once Snowflake itself no longer contains the query history we will be unable to mine it for lineage. Lineage is unsupported for parameterized queries. Snowflake currently does not resolve values for parameterized queries before logging them in query history. This limits Atlan from generating lineage in such cases. When using the account usage extraction method, there are currently some limitations. We are working with Snowflake to find workarounds for crawling the following: External table location data Procedures Primary key designation External table location data Primary key designation Furthermore, only database-level filtering is currently possible. When using the account usage method for fetching metadata, Atlan requires access to the following views in Snowflake: For the crawler: DATABASES , SCHEMATA , TABLES , VIEWS , COLUMNS , and PIPES For the miner and popularity metrics : QUERY_HISTORY , ACCESS_HISTORY , and SESSIONS This error can occur when you're connecting to Snowflake through Okta SSO and enter the URL of your Snowflake instance in a format different from the one used in Okta. Snowflake follows two URL formats: Legacy format - . .snowflakecomputing.com or . . .snowflakecomputing.com New URL format - - .snowflakecomputing.com Ensure that you're using the same Snowflake URL format in Snowflake and Okta. Refer to Snowflake documentation to learn more. If you're getting the following error messages - java.net.UnknownHostException and Name or service not known - this is a known error for users who have upgraded to the Snowflake JDBC driver version 3.13.25., have underscores in their account name, and connect to their Snowflake accounts over private link (for example, https://my_account.us-west-2.privatelink.snowflakecomputing.com ). If your Snowflake account name has an underscore - for example, my_account - the updated JDBC driver will automatically convert underscores to dashes or hyphens - . This does not affect normal URLs because Snowflake accepts URLs with both hyphens and underscores. For private link users, however, the JDBC driver will return an error if there are underscores present in the account name and the connection will fail. To troubleshoot further, refer to Snowflake documentation .


Mine Snowflake | Atlan Documentation

Once you have crawled assets from Snowflake , you can mine its query history to construct lineage. To mine lineage from Snowflake, review the order of operations and then complete the following steps. Select the miner To select the Snowflake miner: In the top right of any screen, navigate to New and then click New Workflow . From the filters along the top, click Miner . From the list of packages, select Snowflake Miner and then click Setup Workflow . Configure the miner To configure the Snowflake miner: For Connection , select the connection to mine. (To select a connection, the crawler must have already run.) For Connection , select the connection to mine. (To select a connection, the crawler must have already run.) For Miner Extraction Method , select Source , Agent , or see the separate instructions for the S3 miner . For Miner Extraction Method , select Source , Agent , or see the separate instructions for the S3 miner . For Snowflake Database : If the connection is configured with access to the snowflake database , choose Default . If the connection can only access a separate cloned database , choose Cloned Database . For Snowflake Database : If the connection is configured with access to the snowflake database , choose Default . If the connection can only access a separate cloned database , choose Cloned Database . If you are using a cloned database, enter the name of the cloned database in Database Name and the name of the cloned schema in Schema Name . If you are using a cloned database, enter the name of the cloned database in Database Name and the name of the cloned schema in Schema Name . For Start time , choose the earliest date from which to mine query history. info ð ª Did you know? The miner restricts you to only querying the past two weeks of query history. If you need to query more history, for example in an initial load, consider using the S3 miner first. After the initial load, you can modify the miner's configuration to use query history extraction. For Start time , choose the earliest date from which to mine query history. ð ª Did you know? The miner restricts you to only querying the past two weeks of query history. If you need to query more history, for example in an initial load, consider using the S3 miner first. After the initial load, you can modify the miner's configuration to use query history extraction. To check for any permissions or other configuration issues before running the miner, click Preflight checks . To check for any permissions or other configuration issues before running the miner, click Preflight checks . At the bottom of the screen, click Next to proceed. At the bottom of the screen, click Next to proceed. Agent extraction method Atlan supports using a Secure Agent for mining query history from Snowflake. To use a Secure Agent, follow these steps: Query redaction is a compute-intensive operation and can impact workflow performance. To use this feature effectively, assign at least 4 CPU cores to the agent. Select the Agent tab. Configure Query Redaction : By default, Query Redaction is turned off. You may choose to configure query redaction: Set Enable Query Redaction flag to true if you want to redact PII data from query history before ingesting it to Atlan. Configure PII Patterns for Query Redaction : Define patterns to identify and redact sensitive information using the following JSON format: { "pii_patterns" : [ { "name" : "email" , "regex" : "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}\\b" , "replacement" : "[EMAIL]" } ] } Set Enable Query Redaction flag to true if you want to redact PII data from query history before ingesting it to Atlan. Configure PII Patterns for Query Redaction : Define patterns to identify and redact sensitive information using the following JSON format: { "pii_patterns" : [ { "name" : "email" , "regex" : "\\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}\\b" , "replacement" : "[EMAIL]" } ] } Configure the Snowflake data source by adding the secret keys for your secret store. For details on the required fields, refer to the connection configuration used when crawling Snowflake . Complete the Secure Agent configuration by following the instructions in the How to configure Secure Agent for workflow execution guide. Click Next after completing the configuration. If running the miner for the first time, Atlan recommends setting a start date around three days prior to the current date and then scheduling it daily to build up to two weeks of query history. Mining two weeks of query history on the first miner run may cause delays. For all subsequent runs, Atlan requires a minimum lag of 24 to 48 hours to capture all the relevant transformations that were part of a session. Learn more about the miner logic here . Configure the miner behavior To configure the Snowflake miner behavior: (Optional) For Calculate popularity , keep True to retrieve usage and popularity metrics for your Snowflake assets from query history. For Excluded Users , type the names of users to be excluded while calculating usage metrics for Snowflake assets. Press Enter after each name to add more names. For Excluded Users , type the names of users to be excluded while calculating usage metrics for Snowflake assets. Press Enter after each name to add more names. (Optional) For Advanced Config , keep Default for the default configuration or click Custom to configure the miner: If Atlan support has provided you with a custom control configuration, enter the configuration into the Custom Config box. You can also enter { ignore-all-case : true} to enable crawling assets with case-sensitive identifiers. For Popularity Window (days) , 90 days is the maximum limit. You can set a shorter popularity window of less than 90 days. If Atlan support has provided you with a custom control configuration, enter the configuration into the Custom Config box. You can also enter { ignore-all-case : true} to enable crawling assets with case-sensitive identifiers. For Popularity Window (days) , 90 days is the maximum limit. You can set a shorter popularity window of less than 90 days. Run the miner To run the Snowflake miner, after completing the configuration steps: To run the miner once immediately, at the bottom of the screen, click the Run button. To schedule the miner to run hourly, daily, weekly, or monthly, at the bottom of the screen, click the Schedule & Run button. Once the miner has completed running, you can see lineage for Snowflake assets that were created in Snowflake between the start time and when the miner ran! ð Select the miner Configure the miner Configure the miner behavior Run the miner


Manage Snowflake tags | Atlan Documentation

Note that object tagging in Snowflake currently requires Enterprise Edition or higher . Atlan enables you to import your Snowflake tags , update your Snowflake assets with the imported tags, and push the tag updates back to Snowflake: Import tags - crawl Snowflake tags from Snowflake to Atlan Reverse sync - sync Snowflake tag updates from Atlan to Snowflake Once you've imported your Snowflake tags to Atlan: Your Snowflake assets in Atlan are automatically enriched with their Snowflake tags. Imported Snowflake tags are mapped to corresponding Atlan tags through case-insensitive name match - multiple Snowflake tags can be matched to a single tag in Atlan. You can also attach Snowflake tags , including tag values, to your Snowflake assets in Atlan - allowing you to categorize your assets at a more granular level. Atlan supports: Allowed values : attach an allowed value from a predefined list of values imported from Snowflake. Tag values: enter any value in Atlan while attaching or editing imported Snowflake tags on an asset. Allowed values : attach an allowed value from a predefined list of values imported from Snowflake. Tag values: enter any value in Atlan while attaching or editing imported Snowflake tags on an asset. You can enable reverse sync to push any tag updates for your Snowflake assets back to Snowflake - including allowed and tag values added to assets in Atlan. You can filter your assets by Snowflake tags and tag and allowed values. Enabling reverse sync only updates existing tags in Snowflake. It neither creates nor deletes any tags in Snowflake. Additional privileges are only required when using the information schema method for fetching metadata. This is because Snowflake stores all tag objects in the ACCOUNT_USAGE schema. If you're using the account usage method to crawl metadata in Atlan or you have configured the Snowflake miner , any permissions required are already set. Account usage method Before you can import tags from Snowflake, you need to do the following: Create tags or have existing tags in Snowflake. Grant the same permissions as required for crawling Snowflake assets to import tags and push updated tags to Snowflake. Information schema method Before you can import tags from Snowflake, you need to do the following: Create tags or have existing tags in Snowflake. Grant additional permissions to import tags from Snowflake. Grant additional permissions to push updated tags to Snowflake. Import Snowflake tags to Atlan You need to be an admin user in Atlan to import Snowflake tags to Atlan. You also need to work with your Snowflake administrator to grant additional permissions to import tags from Snowflake - you may not have access yourself. You can import your Snowflake tags to Atlan through one-way tag sync. The synced Snowflake tags are matched to corresponding tags in Atlan through case-insensitive name match and your Snowflake assets are enriched with their synced tags from Snowflake. To import Snowflake tags to Atlan, you can either: Create a new Snowflake workflow and configure the crawler to import tags. Modify the crawler's configuration for an existing Snowflake workflow to change Import Tags to Yes . If you subsequently modify the workflow to disable tag import, for any tags already imported, Atlan preserves those tags. Once the crawler has completed running, tags imported from Snowflake are available to use for tagging assets! ð View Snowflake tags in Atlan Once you've imported your Snowflake tags, you can view and manage your Snowflake tags in Atlan. To view Snowflake tags: From the left menu of any screen, click Governance . Under the Governance heading of the _Governance cente_r, click Tags . (Optional) Under Tags , click the funnel icon to filter tags by source type. Click Snowflake to filter for tags imported from Snowflake. From the left menu under Tags , select a synced tag - synced tags display the Snowflake ï¸ icon next to the tag name. (Optional) Click the Linked assets tab to view linked assets for your Snowflake tag. (Optional) In the top right, click the pencil icon to add a description and change the tag icon . You can't rename tags synced from Snowflake. Push tag updates to Snowflake Any admin or member user in Atlan can configure reverse sync for tag updates to Snowflake. You also need to work with your Snowflake administrator to grant additional permissions to push updates - you may not have access yourself. Reverse sync is currently only available for imported Snowflake tags in Atlan. The imported tags display a Snowflake ï¸ icon next to the tag name. If using the account usage method , expect a data latency of up to 3 hours for reverse tag sync to be successful. You can enable reverse sync for your imported Snowflake tags in Atlan and push all tag updates for your Snowflake assets back to source. Once you have enabled reverse sync, any Snowflake assets with tags updated in Atlan are also updated in Snowflake. To enable reverse sync for imported Snowflake tags: From the left menu of any screen, click Governance . Under the Governance heading of the _Governance cente_r, click Tags . (Optional) Under Tags , click the funnel icon to filter tags by source type. Click Snowflake to filter for tags imported from Snowflake. In the left menu under Tags , select a synced Snowflake tag - synced tags display the Snowflake ï¸ icon next to the tag name. Under Synced tags , in the upper right, turn on Enable reverse sync to synchronize tag updates from Atlan to Snowflake. In the advanced settings, you can also enable concatenation to support multiple tag values for a single column. For detailed information about multiple tag values and concatenation, see Multiple tag values and concatenation . In the corresponding confirmation dialog, click Yes, enable it to enable reverse tag sync or click Cancel . Now when you attach Snowflake tags to your Snowflake assets in Atlan, these tag updates are also pushed to Snowflake! ð Enabling reverse sync won't trigger any updates in Snowflake until synced tags are attached to Snowflake assets in Atlan. For any questions about managing Snowflake tags, head over here . Import Snowflake tags to Atlan View Snowflake tags in Atlan Push tag updates to Snowflake


Set up Databricks | Atlan Documentation

Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods: Personal access token authentication AWS service principal authentication Azure service principal authentication Personal access token authentication Check that you have Admin and Databricks SQL access for the Databricks workspace. This is required for both cluster options described below. If you don't have this access, contact your Databricks administrator. Grant user access to workspace To grant workspace access to the user creating a personal access token: From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the user. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the user to grant access. For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the user to grant access. For Permission , click the dropdown and select workspace User. Generate a personal access token You can generate a personal access token in your Databricks workspace to the authenticate the integration in Atlan . To generate a personal access token: From the top right of your Databricks workspace, click your Databricks username, and then from the dropdown, click User Settings . Under the Settings menu, click Developer . On the Developer page, next to Access tokens , click Manage . On the Access tokens page, click the Generate new token button. In the Generate new token dialog: For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate . For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate . At the bottom of the dialog, click Generate . Copy and save the generated token in a secure location, and then click Done . Select a cluster Atlan recommends using serverless SQL warehouses for instant compute availability. To enable serverless SQL warehouses, refer to Databricks documentation for AWS Databricks workspaces or Microsoft documentation for Azure Databricks workspaces. You can set up personal access token authentication for your Databricks instance using one of the following cluster options: SQL warehouse (formerly SQL endpoint) To confirm an all-purpose interactive cluster is configured: From the left menu of any page of your Databricks instance, click Compute . Under the All-purpose clusters tab, verify you have a cluster defined. Click the link under the Name column of the table to open your cluster. Under the Configuration tab, verify the Autopilot options to Terminate after . minutes is enabled. At the bottom of the Configuration tab, expand the Advanced options expandable. Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path . Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path . To confirm a SQL warehouse is configured: From the left menu of any page of your Databricks instance, open the dropdown just below the databricks logo and change to SQL . From the refreshed left menu, click SQL Warehouses . Click the link under the Name column of the table to open your SQL warehouse. Under the Connection details tab, confirm that all of the fields are populated and copy them for use in crawling: Server hostname , Port , and HTTP path . AWS service principal authentication You need your AWS Databricks account admin to create a service principal and manage OAuth credentials for the service principal and your AWS Databricks workspace admin to add the service principal to your AWS Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Create a service principal You can create a service principal directly in your Databricks account or from a Databricks workspace. Identity federation enabled on your workspaces: Databricks recommends creating the service principal in the account and assigning it to workspaces. Identity federation disabled on your workspaces: Databricks recommends that you create your service principal from a workspace. To create a service principal from your Databricks account, with identify federation enabled: Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal . On the Add service principal page, enter a name for the service principal and then click Add . Once the service principal has been created, you can assign it to your identity federated workspace. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the service principal you created. For Permission , click the dropdown and select workspace User. To create a service principal from a Databricks workspace, with identity federation disabled: Log in to your AWS Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings . In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access . On the Identity and access page, under Management and permissions , next to Service principals , click Manage . In the upper right of the Service principals page, click Add service principal . In the Add service principal dialog, click the Add new button. For New service principal display name , enter a name for the service principal and then click Add . Create an OAuth secret for the service principal You need to create an OAuth secret to authenticate to Databricks REST APIs. To create an OAuth secret for the service principal : Log in to your Databricks account console as an account admin. Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management . From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, select the service principal you created . In the upper right of the Service principals page, select the service principal you created . On the service principal page, under OAuth secrets , click Generate secret . On the service principal page, under OAuth secrets , click Generate secret . From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. danger Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal. From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal. Once you've copied the client ID and secret, click Done . Once you've copied the client ID and secret, click Done . Azure service principal authentication You need your Azure Databricks account admin to create a service principal and your Azure Databricks workspace admin to add the service principal to your Azure Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Client ID (application ID) Tenant ID (directory ID) Create a service principal To use service principals on Azure Databricks , an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application and then add it to the Azure Databricks workspace to use as a service principal. To create a service principal: Sign in to the Azure portal . If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to create the service principal. Click**+ Add and select App registration**. For_Name_, enter a name for the application. In the_Supported account types_section, select Accounts in this organizational directory only (Single tenant) and then click Register . Application (client) ID Directory (tenant) ID To generate a client secret, within_Manage_, click Certificates & secrets . On the_Client secrets_tab, click New client secret . In the_Add a client secret_dialog, enter the following details: For Description , enter a description for the client secret. For_Expires_, select an expiry time period for the client secret and then click Add . Copy and store the client secret's_Value_in a secure place. Add a service principal to your account To add a service principal to your Azure Databricks account: Log in to your Azure Databricks account console as an account admin. From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal . On the Add service principal page, enter a name for the service principal. Under UUID , paste the Application (client) ID for the service principal. Click Add . Assign a service principal to a workspace To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page. To assign a service principal to your Azure Databricks account: Log in to your Databricks account console as an account admin. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User . For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User . To assign a service principal to your Azure Databricks workspace: Log in to your Azure Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings . In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access . On the Identity and access page, under Management and permissions , next to Service principals , click Manage . In the upper right of the Service principals page, click Add service principal . In the Add service principal dialog, click the Add new button. For New service principal display name , paste the Application (client) ID for the service principal , enter a display name, and then click Add . Grant permissions to crawl metadata You must have a Unity Catalog-enabled Databricks workspace to crawl metadata in Atlan. To extract metadata, you can grant the BROWSE privilege , currently in public preview. You no longer require the Data Reader preset that granted the following privileges on objects in the catalog - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . To grant permissions to a user or service principal: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the left menu of the Catalog Explorer page, select the catalog you want to crawl in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button. In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant . Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant . (Optional) Repeat steps 3-5 for each catalog you want to crawl in Atlan. System tables extraction method To crawl metadata via system tables, you must have a Unity Catalog-enabled workspace and a configured SQL warehouse. Follow these steps to extract metadata using system tables: Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Cross-workspace extraction To crawl metadata from all workspaces within a Databricks metastore using a single connection, see Set up cross-workspace extraction for instructions. (Optional) Grant permissions to query and preview data Atlan currently only supports querying data and viewing sample data preview for the personal access token authentication method. To grant permissions to query data and preview example data: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the left menu of the Catalog Explorer page, select the catalog you want to query and preview data from in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button. In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog. Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . At the bottom of the dialog, click Grant . Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog. Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . At the bottom of the dialog, click Grant . (Optional) Repeat steps 3-5 for each catalog you want to query and preview data from in Atlan. (Optional) Grant permissions to import and update tags To import Databricks tags , you must have a Unity Catalog-enabled workspace and a SQL warehouse configured. Atlan supports importing Databricks tags using system tables for all three authentication methods. Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following privileges: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalog_tags system.information_schema.schema_tags system.information_schema.table_tags system.information_schema.column_tags To push tags updated for assets in Atlan to Databricks, you need to grant the following privileges : APPLY TAG on the object USE CATALOG on the object's parent catalog USE SCHEMA on the object's parent schema (Optional) Grant permissions to extract lineage and usage from system tables You must have a Unity Catalog-enabled workspace to use system tables. Atlan supports extracting the following for your Databricks assets using system tables : usage and popularity metrics Enable system.access schema You need your account admin to enable the system.access schema using the SystemSchemas API . This enables Atlan to extract lineage using system tables. In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore in your Databricks environment for which you want to extract lineage. To verify that system schemas are enabled for each schema, follow the steps in Databricks documentation : List system schemas using the SystemSchemas API to check the status. If enabled for any given schema, the state is EnableCompleted . This confirms that the schema has been enabled for that specific metastore. Atlan can only extract lineage using system tables when the state is marked as EnableCompleted . (Optional) enable system.information_schema.table To generate lineage with the target type set as PATH for a table, Atlan uses metadata from system.information_schema.table to resolve table paths and dependencies. To enable this, you must grant the following permissions on the relevant catalog, schema, and tables. You must be a metastore admin, have the MANAGE privilege on the object, or be the owner of the catalog, schema, or table to grant these permissions. In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore from which you want to extract lineage. Open Catalog Explorer in your Databricks workspace. Navigate to the catalog (for example, main ) and then to the appropriate schema (for example, sales ). Click the Permissions tab. Click Grant . Enter the user or group name (principal). Assign the following permissions: USAGE on the catalog USAGE on the schema SELECT on each relevant table USAGE on the catalog USAGE on the schema SELECT on each relevant table Click Grant to apply the changes. These privileges enable Atlan to read table definitions and other metadata from the metastore. (Optional) enable system.query schema This is only required if you also want to extract usage and popularity metrics from Databricks. You need your account admin to enable the system.query schema using the SystemSchemas API . This enables Atlan to mine query history using system tables for usage and popularity metrics. To verify that system schemas is enabled for each schema, follow the steps in Databricks documentation . If enabled for any given schema, the state is EnableCompleted . ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query ? Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables . Grant permissions Atlan supports extracting Databricks lineage and usage and popularity metrics using system tables for all three authentication methods . Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following permissions: CAN_USE on a SQL warehouse USE_CATALOG on system catalog USE SCHEMA on system.access schema USE SCHEMA on system.query schema (tomine query history for usage and popularity metrics) SELECT on the following tables: system.query.history (to mine query history for usage and popularity metrics) system.access.table_lineage system.access.column_lineage system.query.history (to mine query history for usage and popularity metrics) You need to create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection. ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query ? Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables . (Optional) Create cloned views of system tables When you don't want to grant access to system tables directly, you can create cloned views to expose lineage and popularity metrics through a separate schema. Follow these steps to set up cloned views: Create a catalog and schema to store cloned views. Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema . Create a catalog and schema to store cloned views. Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema . Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW . . column_lineage AS SELECT * FROM system . access . column_lineage ; CREATE OR REPLACE VIEW . . table_lineage AS SELECT * FROM system . access . table_lineage ; Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW . . query_history AS SELECT * FROM system . query . history ; Replace and with the catalog and schema names used in your environment. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW . . column_lineage AS SELECT * FROM system . access . column_lineage ; CREATE OR REPLACE VIEW . . table_lineage AS SELECT * FROM system . access . table_lineage ; Replace and with the catalog and schema names used in your environment. Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW . . query_history AS SELECT * FROM system . query . history ; Replace and with the catalog and schema names used in your environment. Replace and with the catalog and schema names used in your environment. Grant the following permissions to enable access to the cloned views: CAN_USE on a SQL warehouse USE CATALOG on the catalog (for example, ) USE SCHEMA and SELECT on the schema (for example, . ) You must create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection. Locate warehouse ID To extract lineage and usage and popularity metrics using system tables, you will also need the warehouse ID of your SQL warehouse . To locate the warehouse ID: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click SQL Warehouses . On the Compute page, select the warehouse you want to use. (Optional) Grant view permissions to access Databricks entities via APIs Atlan uses Databricks REST APIs to extract metadata for Notebooks, Queries, Jobs, and Pipelines. This information helps to understand which Databricks enitity was used to create a lineage between assets. Use the steps below for each object type to grant CAN VIEW permission to the Databricks user or service principal configured in your integration: Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders . Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders . Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries . Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries . Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job . Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job . Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions . Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions . (Optional) Grant permissions for views and materialized views Atlan requires the following permissions to to extract view definitions from and generate lineagefor views and materialized views: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the Catalog Explorer , select the catalog you want to extract view definitions from and generate lineage for in Atlan. From the tabs at the top, click the Permissions tab, and then click Grant . In the Grant on (workspace name) dialog, configure the following: Select the user or service principal under Principals . Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Select the user or service principal under Principals . Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Click Grant to apply the permissions. Repeat steps 3 6 for each catalog you want to crawl in Atlan. SELECT permission is required to extract the definitions of views and materialized views. If you prefer not to grant SELECT at the catalog level, you can grant it on individual views and materialized views instead. (Optional) Grant permissions to mine query history To mine query history using REST API, you will need to assign the CAN MANAGE permission on your SQL warehouses to the user or service principal. To grant permissions to mine query history: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click SQL Warehouses . On the Compute page, for each SQL warehouse you want to mine query history, click the 3-dot icon and then click Permissions . Expand the Can use permissions dropdown and then select Can manage . This permission enables the service principal to view all queries for the warehouse . Click Add to assign the CAN MANAGE permission to the service principal. Personal access token authentication AWS service principal authentication Azure service principal authentication Grant permissions to crawl metadata (Optional) Grant permissions to query and preview data (Optional) Grant permissions to import and update tags (Optional) Grant permissions to extract lineage and usage from system tables (Optional) Grant view permissions to access Databricks entities via APIs (Optional) Grant permissions for views and materialized views (Optional) Grant permissions to mine query history


Set up Databricks | Atlan Documentation