Atlan | Atlan Documentation

Discover, trust, and govern your data & AI ecosystem Everything you need to get started with Atlan. ð Quick-start guide Step-by-step onboarding ð § Secure agent Enterprise-grade deployment options ð Playbooks automation Rule-based metadata updates at scale ð ¡ï¸ Govern & manage Create data contracts & policies ð Integrate Automation, collaboration & other integrations ï¸ Introductory walkthrough Play with APIs in minutes ð » Client SDKs Java, Python & more ð ¦ Packages Developer-built utilities and integrations Get started with Atlan by building the right strategy and setting a strong foundation. A comprehensive look at Atlan's security philosophy, core values, and rigorous security procedures Help and support Find answers or contact our team for personalized assistance


Atlan - Developers

Are you keen to activate your metadata? To automate tedious manual work, or reduce the complexity of some of your largest data challenges? Welcome to our 🏡 for the developer community of Atlan. Here you'll find reference materials for working with Atlan programmatically. We look forward to helping you, the 🧑's of data, develop the future 🤖's of data!


Atlan | Atlan Documentation


Set up Snowflake | Atlan Documentation

You need your Snowflake administrator to run these commands - you may not have access yourself. Create user and role in Snowflake Create a role and user in Snowflake using the following commands: Create role Create a role in Snowflake using the following commands: Replace with the default warehouse to use when running the Snowflake crawler . Atlan requires the following privileges to: OPERATE enables Atlan to start the virtual warehouse to fetch metadata if the warehouse has stopped. USAGE enables Atlan to show or list metadata from Snowflake. This in turn enables the Snowflake crawler to run the SHOW query. Create a user Create a separate user to integrate into Atlan, using one of the following 3 options: See Snowflake's official guide for details on generating an RSA key-pair . To create a user with a key-pair, replace the value for rsa_public_key with the public key and run the following: Learn more about the SERVICE type property in Snowflake documentation . Atlan only supports encrypted private keys with a non-empty passphrase - generally recommended as more secure. An empty passphrase results in workflow failures. To generate an encrypted private key, omit the -nocrypt option. Refer to Snowflake documentation to learn more. Snowflake recommends transitioning away from basic authentication using username and password. Change to key-pair authentication for enhanced security. For any existing Snowflake workflows, you can modify the crawler configuration to update the authentication method. To create a user with a password, replace and run the following: Learn more about the LEGACY_SERVICE type property in Snowflake documentation . This method is currently only available if Okta is your IdP (Snowflake supports) authenticating natively through Okta : Create a user in your identity provider (IdP) and use federated authentication in Snowflake . The password for this user must be maintained solely in the IdP and multi-factor authentication (MFA) must be disabled. Grant role to user To grant the atlan_user_role to the new user: Configure OAuth (client credentials flow) with Microsoft Entra ID To configure OAuth authentication using Microsoft Entra ID (formerly Azure AD) with the client credentials flow: Follow Snowflake's documentation to: Register a new application in Microsoft Entra ID Collect the client ID , tenant ID , and client secret Add the required API permissions Follow Snowflake's documentation to: Register a new application in Microsoft Entra ID Collect the client ID , tenant ID , and client secret Add the required API permissions In Snowflake, create a security integration using the following: CREATE SECURITY INTEGRATION external_oauth_azure_ad TYPE = external_oauth ENABLED = true EXTERNAL_OAUTH_TYPE = azure EXTERNAL_OAUTH_ISSUER = '\ ' EXTERNAL_OAUTH_JWS_KEYS_URL = '\ ' EXTERNAL_OAUTH_AUDIENCE_LIST = ( '\ ' ) EXTERNAL_OAUTH_TOKEN_USER_MAPPING_CLAIM = 'sub' EXTERNAL_OAUTH_SNOWFLAKE_USER_MAPPING_ATTRIBUTE = 'login_name' ; Replace the placeholders with actual values from your Azure AD app: Your tenant's OAuth 2.0 issuer URL Azure JWKs URI Application ID URI of the Azure app In Snowflake, create a security integration using the following: Replace the placeholders with actual values from your Azure AD app: Your tenant's OAuth 2.0 issuer URL Azure JWKs URI Application ID URI of the Azure app Create a Snowflake user with a login name that exactly matches the Azure AD client object ID: CREATE USER oauth_svc_user WITH LOGIN_NAME = '\ ' -- Use Azure client OBJECT ID DEFAULT_ROLE = \ DEFAULT_WAREHOUSE = \ ; Create a Snowflake user with a login name that exactly matches the Azure AD client object ID: Grant the configured role to this user: GRANT ROLE \ TO USER oauth_svc_user ; Grant the configured role to this user: Choose metadata fetching method Atlan supports two methods for fetching metadata from Snowflake - account usage and information schema. You should choose one of these two methods to set up Snowflake: Grant permissions for account usage method If you want to set warehouse timeouts when using this method, set a large value initially for the workflow to succeed. Once the workflow has succeeded, adjust the value to be twice the extraction time. This method uses the views in SNOWFLAKE.ACCOUNT_USAGE (or a copied version of this schema) to fetch the metadata from Snowflake into Atlan. You can be more granular with permissions using this method, but there are limitations with this approach . To crawl assets, generate lineage, and import tags If you also want to be able to preview and query the data, you can use the preview and query existing assets permissions instead. Snowflake stores all tag objects in the ACCOUNT_USAGE schema. If you're using the account usage method to crawl metadata in Atlan or you have configured the Snowflake miner , you need to grant the same permissions to import tags as required for crawling Snowflake assets. Note that object tagging in Snowflake currently requires Enterprise Edition or higher . To use the default SNOWFLAKE database and ACCOUNT_USAGE schema and also mine Snowflake's query history (for lineage), grant these permissions: The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: Replace with the copied Snowflake database name. Replace with the copied Snowflake ACCOUNT_USAGE schema name. The grants for the copied version can't be used on the original SNOWFLAKE database. This is because Snowflake produces an error that granular grants can't be given to imported databases. When using a cloned or copied version, verify that the table or view definition remains unchanged as in your SNOWFLAKE database. If the format is different. For example, a column is missing and it no longer qualifies as a clone. To crawl streams To crawl streams, provide the following permissions: To crawl current streams: Replace with the Snowflake database name. Replace with the Snowflake database name. To crawl future streams: To crawl future streams: Replace with the Snowflake database name. (Optional) To preview and query existing assets To query and preview data within assets that already exist in Snowflake, add these permissions: Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you wish to preview and query in Atlan.) (Optional) To preview and query future assets To query and preview data within assets that may be created in the future in Snowflake, add these permissions. Again, if you want to also be able to preview and query the data for future assets, you can add the preview and query future assets permissions instead. Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you want to preview and query in Atlan.) Verify that all the assets you'd like to crawl are present in these grants by checking the grants on the user role defined for the crawler. Grant permissions for information schema method This method uses views in the INFORMATION_SCHEMA schema in Snowflake databases to fetch metadata. You still need to grant specific permissions to enable Atlan to crawl metadata, preview data, and query data with this method. To crawl existing assets Grant these permissions to crawl assets that already exist in Snowflake. If you also want to be able to preview and query the data, you can use the preview and query existing assets permissions instead. Grant permissions to crawl existing assets: Replace with the database you want to be available in Atlan. (Repeat the statements for every database you wish to integrate into Atlan.) Grant permissions to crawl functions: Replace with the database you want to be available in Atlan. (Repeat the statements for every database you wish to integrate into Atlan.) For secure user-defined functions (UDFs), grant OWNERSHIP permissions to retrieve metadata: Replace the placeholders with the appropriate values: : The name of the schema that contains the user-defined function (UDF). : The name of the secure UDF that requires ownership permissions. : The role that gets assigned ownership of the secure UDF. The statements given on this page apply to all schemas, tables, and views in a database in Snowflake. If you want to limit access to only certain objects, you can instead specify the exact objects individually as well. To crawl future assets To crawl assets that may be created in the future in Snowflake, add these permissions. Again, if you want to also be able to preview and query the data for future assets, you can add the preview and query future assets permissions instead. To grant permissions at a database level: Replace with the database you want to crawl in Atlan. (Repeat the statements for every database you want to integrate into Atlan.) For any future grants defined at a schema level to a different role, the schema-level grants take precedence over the database-level grants and any future grants defined at a database level to the Atlan role get ignored. To learn more, refer to Snowflake documentation . To grant permissions at a schema level: Replace with the database and with the schema you want to crawl in Atlan. (Repeat the statements for every database and schema you want to integrate into Atlan.) To mine query history for lineage To also mine Snowflake's query history (for lineage), add these permissions. You can use either option: To mine query history direct from Snowflake's internal tables: To mine query history from a cloned or copied set of tables, where you can also remove any sensitive data: Replace with the name of the cloned database, and with the name of the cloned schema containing account usage details. When using a cloned or copied version, verify that the table or view definition remains unchanged as in your SNOWFLAKE database. If the format is different. For example, a column is missing and it no longer qualifies as a clone. (Optional) To preview and query existing assets To query and preview data within assets that already exist in Snowflake, add these permissions: Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you wish to preview and query in Atlan.) (Optional) To preview and query future assets To query and preview data within assets that may be created in the future in Snowflake, add these permissions. Again, if you want to also be able to preview and query the data for future assets, you can add the preview and query future assets permissions instead. Replace with the database you want to be able to preview and query in Atlan. (Repeat the statements for every database you want to preview and query in Atlan.) For any future grants defined at a schema level to a different role, the schema-level grants take precedence over the database-level grants and any future grants defined at a database level to the Atlan role get ignored. To learn more, refer to Snowflake documentation . To grant permissions at a schema level: Replace with the database and with the schema you want to be able to preview and query in Atlan. (Repeat the statements for every database and schema you want to preview and query in Atlan.) Verify that all the assets you'd like to crawl are present in these grants by checking the grants on the user role defined for the crawler. (Optional) To import Snowflake tags Snowflake stores all tag objects in the ACCOUNT_USAGE schema. Note that object tagging in Snowflake currently requires Enterprise Edition or higher . To import tags from Snowflake , grant these permissions: To use the default SNOWFLAKE database and ACCOUNT_USAGE schema and also mine Snowflake's query history (for lineage), grant these permissions: The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. The ACCOUNTADMIN role is required to grant privileges on the SNOWFLAKE database due to the following reasons: By default, only the ACCOUNTADMIN role can access the SNOWFLAKE database. To enable other roles to access the database and schemas and query the views, a user with the ACCOUNTADMIN role needs to grant IMPORTED PRIVILEGES on the SNOWFLAKE database to the desired roles. To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: To use a copied or cloned version of this default schema, where you can also remove any sensitive data for security purposes, grant these permissions: Replace with the copied Snowflake database name. Replace with the copied Snowflake ACCOUNT_USAGE schema name. The grants for the copied version can't be used on the original SNOWFLAKE database. This is because Snowflake produces an error that granular grants can't be given to imported databases. (Optional) To push updated tags to Snowflake To push tags updated for assets in Atlan to Snowflake , grant these permissions: You can learn more about tag privileges from Snowflake documentation . (Optional) To crawl dynamic tables Atlan currently supports fetching metadata for dynamic tables using the MONITOR privilege. Refer to Snowflake documentation to learn more. To crawl existing dynamic tables from Snowflake: Grant permissions at a database level: Grant permissions at a schema level: To crawl future dynamic tables from Snowflake: Grant permissions at a database level: Grant permissions at a schema level: Replace with the database and with the schema you want to crawl in Atlan. (Repeat the statements for every database and schema you want to integrate into Atlan.) (Optional) To crawl Iceberg tables Atlan currently supports fetching metadata for Iceberg tables only for the information schema extraction method. To crawl Iceberg tables from Snowflake, grant the following permissions: To crawl existing Iceberg tables in Snowflake: To crawl future Iceberg tables in Snowflake: To crawl Iceberg catalog metadata for Iceberg tables in Snowflake: You must first grant permissions to crawl existing Iceberg tables for this permission to work on catalogs. You must also grant permissions to all the catalogs you want to crawl in Atlan individually. (Optional) To crawl Snowflake stages Atlan supports crawling metadata for Snowflake stages using the USAGE and READ privileges. For more information, see the Snowflake documentation for INFORMATION_SCHEMA.STAGES. To crawl stages from Snowflake: Grant USAGE and READ privileges on all existing stages at the database level: Replace with the name of your Snowflake database Replace with the name of your Snowflake database Replace with the role you've granted Atlan to use for crawling. Replace with the role you've granted Atlan to use for crawling. Grant USAGE and READ privileges on all future stages at the database level: Grant USAGE and READ privileges on all future stages at the database level: Replace with the name of your Snowflake database Replace with the role you've granted Atlan to use for crawling. Allowlist the Atlan IP If you are using the IP allowlist in your Snowflake instance, you must add the Atlan IP to the allowlist. Please raise a support ticket from within Atlan, or submit a request . (If you aren't using the IP allowlist in your Snowflake instance, you can skip this step.) Create user and role in Snowflake Choose metadata fetching method Grant permissions for account usage method Grant permissions for information schema method Allowlist the Atlan IP


Set up Databricks | Atlan Documentation

Atlan supports three authentication methods for fetching metadata from Databricks. You can set up any of the following authentication methods: Personal access token authentication AWS service principal authentication Azure service principal authentication Personal access token authentication Check that you have Admin and Databricks SQL access for the Databricks workspace. This is required for both cluster options described below. If you don't have this access, contact your Databricks administrator. Grant user access to workspace To grant workspace access to the user creating a personal access token: From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the user. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the user to grant access. For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the user to grant access. For Permission , click the dropdown and select workspace User. Generate a personal access token You can generate a personal access token in your Databricks workspace to the authenticate the integration in Atlan . To generate a personal access token: From the top right of your Databricks workspace, click your Databricks username, and then from the dropdown, click User Settings . Under the Settings menu, click Developer . On the Developer page, next to Access tokens , click Manage . On the Access tokens page, click the Generate new token button. In the Generate new token dialog: For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate . For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Comment , enter a description of the token's intended use - for example, Atlan crawler . For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. Important! If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. For Lifetime (days) , consider removing the number. This enables the token to be used indefinitely - it won't need to be refreshed. If you do enter a number, remember that you need to periodically regenerate it and update Atlan's crawler configuration with the new token each time. At the bottom of the dialog, click Generate . At the bottom of the dialog, click Generate . Copy and save the generated token in a secure location, and then click Done . Select a cluster Atlan recommends using serverless SQL warehouses for instant compute availability. To enable serverless SQL warehouses, refer to Databricks documentation for AWS Databricks workspaces or Microsoft documentation for Azure Databricks workspaces. You can set up personal access token authentication for your Databricks instance using one of the following cluster options: SQL warehouse (formerly SQL endpoint) To confirm an all-purpose interactive cluster is configured: From the left menu of any page of your Databricks instance, click Compute . Under the All-purpose clusters tab, verify you have a cluster defined. Click the link under the Name column of the table to open your cluster. Under the Configuration tab, verify the Autopilot options to Terminate after . minutes is enabled. At the bottom of the Configuration tab, expand the Advanced options expandable. Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path . Under the Advanced options expandable, open the JDBC/ODBC tab. Confirm that all of the fields in this tab are populated, and copy them for use in crawling: Server Hostname , Port , and HTTP Path . To confirm a SQL warehouse is configured: From the left menu of any page of your Databricks instance, open the dropdown just below the databricks logo and change to SQL . From the refreshed left menu, click SQL Warehouses . Click the link under the Name column of the table to open your SQL warehouse. Under the Connection details tab, confirm that all of the fields are populated and copy them for use in crawling: Server hostname , Port , and HTTP path . AWS service principal authentication You need your AWS Databricks account admin to create a service principal and manage OAuth credentials for the service principal and your AWS Databricks workspace admin to add the service principal to your AWS Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Create a service principal You can create a service principal directly in your Databricks account or from a Databricks workspace. Identity federation enabled on your workspaces: Databricks recommends creating the service principal in the account and assigning it to workspaces. Identity federation disabled on your workspaces: Databricks recommends that you create your service principal from a workspace. To create a service principal from your Databricks account, with identify federation enabled: Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal . On the Add service principal page, enter a name for the service principal and then click Add . Once the service principal has been created, you can assign it to your identity federated workspace. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown and select workspace User. For User, group, or service principal , select the service principal you created. For Permission , click the dropdown and select workspace User. To create a service principal from a Databricks workspace, with identity federation disabled: Log in to your AWS Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings . In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access . On the Identity and access page, under Management and permissions , next to Service principals , click Manage . In the upper right of the Service principals page, click Add service principal . In the Add service principal dialog, click the Add new button. For New service principal display name , enter a name for the service principal and then click Add . Create an OAuth secret for the service principal You need to create an OAuth secret to authenticate to Databricks REST APIs. To create an OAuth secret for the service principal : Log in to your Databricks account console as an account admin. Log in to your Databricks account console as an account admin. From the left menu of the account console, click User management . From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, select the service principal you created . In the upper right of the Service principals page, select the service principal you created . On the service principal page, under OAuth secrets , click Generate secret . On the service principal page, under OAuth secrets , click Generate secret . From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. danger Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal. From the Generate secret dialog, copy the Secret and Client ID and store it in a secure location. Note that this secret is only revealed once during creation. The client ID is the same as the application ID of the service principal. Once you've copied the client ID and secret, click Done . Once you've copied the client ID and secret, click Done . Azure service principal authentication You need your Azure Databricks account admin to create a service principal and your Azure Databricks workspace admin to add the service principal to your Azure Databricks workspace - you may not have access yourself. You need the following to authenticate the connection in Atlan: Client ID (application ID) Tenant ID (directory ID) Create a service principal To use service principals on Azure Databricks , an admin user must create a new Microsoft Entra ID (formerly Azure Active Directory) application and then add it to the Azure Databricks workspace to use as a service principal. To create a service principal: Sign in to the Azure portal . If you have access to multiple tenants, subscriptions, or directories, click the Directories + subscriptions (directory with filter) icon in the top menu to switch to the directory in which you want to create the service principal. Click**+ Add and select App registration**. For_Name_, enter a name for the application. In the_Supported account types_section, select Accounts in this organizational directory only (Single tenant) and then click Register . Application (client) ID Directory (tenant) ID To generate a client secret, within_Manage_, click Certificates & secrets . On the_Client secrets_tab, click New client secret . In the_Add a client secret_dialog, enter the following details: For Description , enter a description for the client secret. For_Expires_, select an expiry time period for the client secret and then click Add . Copy and store the client secret's_Value_in a secure place. Add a service principal to your account To add a service principal to your Azure Databricks account: Log in to your Azure Databricks account console as an account admin. From the left menu of the account console, click User management . From the tabs along the top of the User management page, click the Service principals tab. In the upper right of the Service principals page, click Add service principal . On the Add service principal page, enter a name for the service principal. Under UUID , paste the Application (client) ID for the service principal. Click Add . Assign a service principal to a workspace To add users to a workspace using the account console, the workspace must be enabled for identity federation. Workspace admins can also assign service principals to workspaces using the workspace admin settings page. To assign a service principal to your Azure Databricks account: Log in to your Databricks account console as an account admin. From the left menu of the account console, click Workspaces and then select a workspace to which you want to add the service principal. From the tabs along the top of your workspace page, click the Permissions tab. In the upper right of the Permissions page, click Add permissions . In the Add permissions dialog, enter the following details: For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User . For User, group, or service principal , select the service principal you created. For Permission , click the dropdown to select workspace User . To assign a service principal to your Azure Databricks workspace: Log in to your Azure Databricks workspace as a workspace admin. From the top right of your workspace, click your username, and then from the dropdown, click Admin Settings . In the left menu of the Settings page, under the Workspace admin subheading, click Identity and access . On the Identity and access page, under Management and permissions , next to Service principals , click Manage . In the upper right of the Service principals page, click Add service principal . In the Add service principal dialog, click the Add new button. For New service principal display name , paste the Application (client) ID for the service principal , enter a display name, and then click Add . Grant permissions to crawl metadata You must have a Unity Catalog-enabled Databricks workspace to crawl metadata in Atlan. To extract metadata, you can grant the BROWSE privilege , currently in public preview. You no longer require the Data Reader preset that granted the following privileges on objects in the catalog - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . To grant permissions to a user or service principal: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the left menu of the Catalog Explorer page, select the catalog you want to crawl in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button. In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant . Under Principals , click the dropdown and then select the user or service principal. Under Privileges , check the BROWSE privilege. At the bottom of the dialog, click Grant . (Optional) Repeat steps 3-5 for each catalog you want to crawl in Atlan. System tables extraction method To crawl metadata via system tables, you must have a Unity Catalog-enabled workspace and a configured SQL warehouse. Follow these steps to extract metadata using system tables: Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Create one of the following authentication methods: Personal access token AWS service principal Azure service principal Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Grant the following privileges to the identity you created: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalogs system.information_schema.schemata system.information_schema.tables system.information_schema.columns system.information_schema.key_column_usage system.information_schema.table_constraints Cross-workspace extraction To crawl metadata from all workspaces within a Databricks metastore using a single connection, see Set up cross-workspace extraction for instructions. (Optional) Grant permissions to query and preview data Atlan currently only supports querying data and viewing sample data preview for the personal access token authentication method. To grant permissions to query data and preview example data: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the left menu of the Catalog Explorer page, select the catalog you want to query and preview data from in Atlan. From the tabs along the top of your workspace page, click the Permissions tab and then click the Grant button. In the Grant on (workspace name) dialog, configure the following: Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog. Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . At the bottom of the dialog, click Grant . Under Principals , click the dropdown and then select the user or service principal. Under Privilege presets , click the dropdown and then click Data Reader to enable read-only access to the catalog. Doing so automatically selects the following privileges - USE CATALOG , USE SCHEMA , EXECUTE , READ VOLUME , and SELECT . At the bottom of the dialog, click Grant . (Optional) Repeat steps 3-5 for each catalog you want to query and preview data from in Atlan. (Optional) Grant permissions to import and update tags To import Databricks tags , you must have a Unity Catalog-enabled workspace and a SQL warehouse configured. Atlan supports importing Databricks tags using system tables for all three authentication methods. Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following privileges: CAN_USE on a SQL warehouse USE CATALOG on system catalog USE SCHEMA on system.information_schema SELECT on the following tables: system.information_schema.catalog_tags system.information_schema.schema_tags system.information_schema.table_tags system.information_schema.column_tags To push tags updated for assets in Atlan to Databricks, you need to grant the following privileges : APPLY TAG on the object USE CATALOG on the object's parent catalog USE SCHEMA on the object's parent schema (Optional) Grant permissions to extract lineage and usage from system tables You must have a Unity Catalog-enabled workspace to use system tables. Atlan supports extracting the following for your Databricks assets using system tables : usage and popularity metrics Enable system.access schema You need your account admin to enable the system.access schema using the SystemSchemas API . This enables Atlan to extract lineage using system tables. In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore in your Databricks environment for which you want to extract lineage. To verify that system schemas are enabled for each schema, follow the steps in Databricks documentation : List system schemas using the SystemSchemas API to check the status. If enabled for any given schema, the state is EnableCompleted . This confirms that the schema has been enabled for that specific metastore. Atlan can only extract lineage using system tables when the state is marked as EnableCompleted . (Optional) enable system.information_schema.table To generate lineage with the target type set as PATH for a table, Atlan uses metadata from system.information_schema.table to resolve table paths and dependencies. To enable this, you must grant the following permissions on the relevant catalog, schema, and tables. You must be a metastore admin, have the MANAGE privilege on the object, or be the owner of the catalog, schema, or table to grant these permissions. In Atlan, one Databricks connection corresponds to one metastore. Repeat the following process for each metastore from which you want to extract lineage. Open Catalog Explorer in your Databricks workspace. Navigate to the catalog (for example, main ) and then to the appropriate schema (for example, sales ). Click the Permissions tab. Click Grant . Enter the user or group name (principal). Assign the following permissions: USAGE on the catalog USAGE on the schema SELECT on each relevant table USAGE on the catalog USAGE on the schema SELECT on each relevant table Click Grant to apply the changes. These privileges enable Atlan to read table definitions and other metadata from the metastore. (Optional) enable system.query schema This is only required if you also want to extract usage and popularity metrics from Databricks. You need your account admin to enable the system.query schema using the SystemSchemas API . This enables Atlan to mine query history using system tables for usage and popularity metrics. To verify that system schemas is enabled for each schema, follow the steps in Databricks documentation . If enabled for any given schema, the state is EnableCompleted . ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query ? Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables . Grant permissions Atlan supports extracting Databricks lineage and usage and popularity metrics using system tables for all three authentication methods . Once you have created a personal access token , an AWS service principal , or an Azure service principal , you will need to grant the following permissions: CAN_USE on a SQL warehouse USE_CATALOG on system catalog USE SCHEMA on system.access schema USE SCHEMA on system.query schema (tomine query history for usage and popularity metrics) SELECT on the following tables: system.query.history (to mine query history for usage and popularity metrics) system.access.table_lineage system.access.column_lineage system.query.history (to mine query history for usage and popularity metrics) You need to create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection. ð ª Did you know? Can't grant SELECT permissions on the system tables in system.access and system.query ? Skip the previous steps and create cloned views in a separate catalog and schema. See Create cloned views of system tables . (Optional) Create cloned views of system tables When you don't want to grant access to system tables directly, you can create cloned views to expose lineage and popularity metrics through a separate schema. Follow these steps to set up cloned views: Create a catalog and schema to store cloned views. Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema . Create a catalog and schema to store cloned views. Use meaningful and unique names for example, atlan_cloned_catalog and atlan_cloned_schema . Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW . . column_lineage AS SELECT * FROM system . access . column_lineage ; CREATE OR REPLACE VIEW . . table_lineage AS SELECT * FROM system . access . table_lineage ; Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW . . query_history AS SELECT * FROM system . query . history ; Replace and with the catalog and schema names used in your environment. Create cloned views for the following system tables: Lineage tables CREATE OR REPLACE VIEW . . column_lineage AS SELECT * FROM system . access . column_lineage ; CREATE OR REPLACE VIEW . . table_lineage AS SELECT * FROM system . access . table_lineage ; Replace and with the catalog and schema names used in your environment. Replace and with the catalog and schema names used in your environment. Popularity metrics CREATE OR REPLACE VIEW . . query_history AS SELECT * FROM system . query . history ; Replace and with the catalog and schema names used in your environment. Replace and with the catalog and schema names used in your environment. Grant the following permissions to enable access to the cloned views: CAN_USE on a SQL warehouse USE CATALOG on the catalog (for example, ) USE SCHEMA and SELECT on the schema (for example, . ) You must create a Databricks connection in Atlan for each metastore. You can use the hostname of your Unity Catalog-enabled workspace as the Host for the connection. Locate warehouse ID To extract lineage and usage and popularity metrics using system tables, you will also need the warehouse ID of your SQL warehouse . To locate the warehouse ID: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click SQL Warehouses . On the Compute page, select the warehouse you want to use. (Optional) Grant view permissions to access Databricks entities via APIs Atlan uses Databricks REST APIs to extract metadata for Notebooks, Queries, Jobs, and Pipelines. This information helps to understand which Databricks enitity was used to create a lineage between assets. Use the steps below for each object type to grant CAN VIEW permission to the Databricks user or service principal configured in your integration: Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders . Notebook API ( /api/2.0/workspace/list ): Grant CAN VIEW permission on individual notebooks, or on the workspace folder containing the notebooks, or on the entire workspace. For more information, see Manage Access Control Lists with Folders . Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries . Queries API ( /api/2.0/sql/queries ): Grant CAN VIEW permission on individual queries, or on the workspace folder containing the queries, or on the entire workspace. For more information, see View Queries . Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job . Job API ( /api/2.2/jobs/list ): Grant CAN VIEW permission on each job object directly. Databricks Jobs are distinct from notebooks or files and require permission set directly on the job object. For more information, see Control Access to a Job . Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions . Pipeline API ( /api/2.0/pipelines ): Grant CAN VIEW permission on each Delta Live Tables (DLT) pipeline object directly. For more information, see Configure Pipeline Permissions . (Optional) Grant permissions for views and materialized views Atlan requires the following permissions to to extract view definitions from and generate lineagefor views and materialized views: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click Catalog . In the Catalog Explorer , select the catalog you want to extract view definitions from and generate lineage for in Atlan. From the tabs at the top, click the Permissions tab, and then click Grant . In the Grant on (workspace name) dialog, configure the following: Select the user or service principal under Principals . Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Select the user or service principal under Principals . Select the following privileges under Privilege presets : USE CATALOG USE SCHEMA SELECT Click Grant to apply the permissions. Repeat steps 3 6 for each catalog you want to crawl in Atlan. SELECT permission is required to extract the definitions of views and materialized views. If you prefer not to grant SELECT at the catalog level, you can grant it on individual views and materialized views instead. (Optional) Grant permissions to mine query history To mine query history using REST API, you will need to assign the CAN MANAGE permission on your SQL warehouses to the user or service principal. To grant permissions to mine query history: Log in to your Databricks workspace as a workspace admin. From the left menu of your workspace, click SQL Warehouses . On the Compute page, for each SQL warehouse you want to mine query history, click the 3-dot icon and then click Permissions . Expand the Can use permissions dropdown and then select Can manage . This permission enables the service principal to view all queries for the warehouse . Click Add to assign the CAN MANAGE permission to the service principal. Personal access token authentication AWS service principal authentication Azure service principal authentication Grant permissions to crawl metadata (Optional) Grant permissions to query and preview data (Optional) Grant permissions to import and update tags (Optional) Grant permissions to extract lineage and usage from system tables (Optional) Grant view permissions to access Databricks entities via APIs (Optional) Grant permissions for views and materialized views (Optional) Grant permissions to mine query history


Set up Microsoft Power BI | Atlan Documentation

Depending on the authentication method you choose, you may need a combination of your Cloud Application Administrator or Application Administrator for Microsoft Entra ID, Microsoft 365 administrator for Microsoft 365, and Fabric Administrator ( formerly known as Power BI Administrator ) for Microsoft Power BI to complete these tasks -> you may not have access yourself. This guide outlines how to set up Microsoft Power BI so it can connect with Atlan for metadata extraction and lineage tracking. Before you begin Register application in Microsoft Entra ID You need your Cloud Application Administrator or Application Administrator to complete these steps > you may not have access yourself. This is required if the creation of registered applications isn't enabled for the entire organization. To register a new application in Microsoft Entra ID: Log in to the Azure portal . Click App registrations from the left menu. Click + New registration . Enter a name for your client application and click Register . Application (client) ID Directory (tenant) ID Click Certificates & secrets from the left menu. Under Client secrets , click + New client secret . Enter a description, select an expiry time, and click Add . Copy and securely store the client secret Value . Create security group in Microsoft Entra ID You need your Cloud Application Administrator or Application Administrator to complete these steps - you may not have access yourself. To create a security group for your application: Log in to the Azure portal . Click Groups under the Manage section. Click New group . Set the Group type to Security . Enter a Group name and optional description. Click No members selected . Click Select and then Create . By the end of these steps, you have registered an application with Microsoft Entra ID and created a Security Group with the appropriate member. Configure authentication options Atlan supports two authentication methods for fetching metadata from Microsoft Power BI: Service principal authentication (recommended) When using Service Principal authentication, you must decide how the connector shall access metadata to catalog assets and build lineage. There are two supported options: This option grants permissions that let the service principal to access only admin-level Power BI APIs. In this mode, Atlan extracts metadata exclusively using administrative endpoints. This option is recommended for stricter access control environments. You need your Fabric Administrator ( formerly known as Power BI Administrator ) to complete these tasks - you may not have access yourself. To configure admin API access: Log in to the Power BI admin portal . Click Tenant settings under Admin portal. Under Admin API settings : Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group under Specific security groups Click Apply Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group under Specific security groups Click Apply Add your security group under Specific security groups Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Add your security group This option grants permissions that let the service principal to access both admin and non-admin Power BI APIs. This enables Atlan to extract richer metadata and build detailed lineage across Power BI assets. You need to be at least a member of the Microsoft Power BI workspace to which you want to add the security group to complete these steps - you may not have access yourself. Make sure that you add the security group from the homepage and not the admin portal. To assign a Microsoft Power BI workspace role to the security group: Open the Microsoft Power BI homepage . Open Workspaces and select the workspace you want to access from Atlan. Click Access . In the panel: Enter the name of your security group where it says Enter email addresses Choose one of the following roles: Viewer : For workspaces without parameters Contributor : For workspaces with semantic models containing parameters or to generate lineage for measures Member : To generate lineage for dataflows Click Add . Enter the name of your security group where it says Enter email addresses Choose one of the following roles: Viewer : For workspaces without parameters Contributor : For workspaces with semantic models containing parameters or to generate lineage for measures Member : To generate lineage for dataflows Viewer : For workspaces without parameters Contributor : For workspaces with semantic models containing parameters or to generate lineage for measures Member : To generate lineage for dataflows Click Add . You need your Fabric Administrator ( formerly known as Power BI Administrator ) to complete these tasks - you may not have access yourself. To enable both admin and non-admin API access: Log in to the Power BI admin portal . Click Tenant settings under Admin portal. Under Developer settings : Expand Service principals can use Fabric APIs and set to Enabled Add your security group under Specific security groups Click Apply Expand Service principals can use Fabric APIs and set to Enabled Add your security group under Specific security groups Click Apply Add your security group under Specific security groups Under Admin API settings : Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Expand Enable service principals to use read-only Power BI admin APIs and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply Add your security group After making these changes, you typically need to wait 15-30 minutes for the settings to take effect across Microsoft's services. Delegated user authentication Atlan doesn't recommend using delegated user authentication as it's also not recommended by Microsoft. You need your Microsoft 365 administrator to complete these steps - you may not have access yourself. To assign the delegated user to the Fabric Administrator role: Open the Microsoft 365 admin portal . Click Users and then Active users from the left menu. Select the delegated user. Under Roles , click Manage roles . Expand Show all by category . Under Collaboration , select Fabric Administrator . Click Save changes . You need your Cloud Application Administrator or Application Administrator to complete these steps, you may not have access yourself. The following permissions are only required for delegated user authentication. If using service principal authentication, you don't need to configure any delegated permissions for a service principal it's recommended that you avoid adding these permissions. These are never used and can cause errors that may be hard to troubleshoot. To add permissions for the registered application : In your app registration, click API permissions under the Manage section. Click Add a permission . Click Delegated permissions and select: Capacity.Read.All Dataflow.Read.All Dataset.Read.All Report.Read.All Tenant.Read.All Workspace.Read.All Click Grant Admin consent (If you only see the Add permissions button, you aren't an administrator). You need your Fabric Administrator ( formerly known as Power BI Administrator ) to complete these tasks, you may not have access yourself. To enable the Microsoft Power BI admin API: Log in to the Power BI admin portal . Click Tenant settings under Admin portal. Under Admin API settings : Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply . Expand Enhance admin APIs responses with detailed metadata and set to Enabled Add your security group Click Apply Add your security group Expand Enhance admin APIs responses with DAX and mashup expressions and set to Enabled Add your security group Click Apply . Add your security group Click Apply . Before you begin Configure authentication options


Atlan architecture | Atlan Documentation

Atlan is a cloud-first solution. Single-tenant SaaS is the recommended deployment model. Atlan currently supports hosting tenants on the following cloud platforms: Amazon Web Services (AWS) Microsoft Azure Google Cloud Platform (GCP) The components of Atlan are isolated, across both compute and data. For more details, see How are resources isolated? Platform components Kong is an API gateway. It handles rate limiting and token verification on all incoming API requests. Apache Keycloak is an identity and access management component. It manages everything to do with users, login, SSO and so on. Heracles is Atlan's API service. It houses the business logic used by the frontend and APIs to interact with other platform components. PostgreSQL is a SQL database. Many services on the platform use it for storage. HashiCorp Vault is a secret manager. It stores sensitive credentials provided by the user. Apache Ranger is the policy engine. It provides fine-grained access control over data in the metastore. Argo Workflows is a workflow orchestrator for k8s. It runs and manages long-running jobs in a container and k8s-native fashion. Admission Controller is a k8s admission controller. It performs certain actions when Argo Workflows are updated such as workflow alerts. Apache Zookeeper manages consensus and coordination for the metastore services. Apache Cassandra is an object-oriented database used to store the metastore's data. Apache Kafka is an event stream. It enables event-driven use cases across the platform. Heka is Atlan's SQL component. It parses, rewrites and optimizes SQL queries and is powered by Apache Calcite . Redis is a cache layer used by Heracles. Platform management components Velero performs cluster backups. Fluent Bit is a logging and metrics processor. It parses and pushes logs from pods to various destinations. Central components Zenduty is used for incident response. Alerts are sent when something goes wrong in one of the clusters. Argo CD is used for continuous deployment. Changes in git repositories lead to upgrades in the clusters. Github Actions update the Docker container images as part of the development process. Sendgrid is used to send emails. The frontend is a Vue.js web application that's hosted on S3 and delivered via Amazon CloudFront content delivery network (CDN) service. Alertmanager sends alerts generated by metrics stored in Prometheus. Grafana provides observability dashboards. VictoriaMetrics is a fast, cost-effective, and scalable monitoring solution and time series database. It processes high volumes of data and enables long-term storing. Atlan marketplace (not pictured) The marketplace offers packages (workflows) that perform long-running tasks on the Atlan platform. The ecosystem enables the creation of metadata and lineage connectors. See security.atlan.com for the latest policies and standards, reports and certifications, architecture, diagrams and more. Platform management components Atlan marketplace (not pictured)


Use the Atlan browser extension | Atlan Documentation

The Atlan browser extension provides metadata context directly in your supported data tools . You can use the extension in the following Chromium-based browsers: Google Chrome and Microsoft Edge. Install the extension To install Atlan's browser extension: You can either: Find the extension in the Chrome Web Store: https://chrome.google.com/webstore/detail/atlan/fipjfjlalpnbejlmmpfnmlkadjgaaheg From the upper right of any screen in Atlan, navigate to your name and then click Profile . Click the four dots icon in the resulting dialog to get to integrations. Under Apps , for Browser extension , click Install . Find the extension in the Chrome Web Store: https://chrome.google.com/webstore/detail/atlan/fipjfjlalpnbejlmmpfnmlkadjgaaheg From the upper right of any screen in Atlan, navigate to your name and then click Profile . Click the four dots icon in the resulting dialog to get to integrations. Under Apps , for Browser extension , click Install . Click the four dots icon in the resulting dialog to get to integrations. Under Apps , for Browser extension , click Install . To install the Atlan browser extension: For Google Chrome, in the upper right of your screen, click Add to Chrome . When prompted for confirmation, click the Add extension button. For Microsoft Edge, follow the steps in Add an extension to Microsoft Edge from the Chrome Web Store . For Google Chrome, in the upper right of your screen, click Add to Chrome . When prompted for confirmation, click the Add extension button. For Microsoft Edge, follow the steps in Add an extension to Microsoft Edge from the Chrome Web Store . Currently, you can't install the browser extension on mobile devices or tablets. You can also install Atlan's browser extension at the workspace level . To set this up, you need to be an administrator or have access to the admin console of your organization's Google account. If your organization uses managed browsers, you can configure the extension for managed browsers . Configure the extension Once installed, configure the Atlan browser extension to get started. Optionally, Atlan admins can preconfigure custom domains for data sources , if any. Configure the extension as a user To configure the browser extension, once installed: If you are logged into your Atlan instance, skip to the next step. If you haven't logged into Atlan, log in to your Atlan instance when prompted. In the Options page, to enter the URL of your Atlan instance: If your organization uses an Atlan domain (for example, _mycompany_.atlan.com ), the Atlan instance URL appears preselected. Click Get started . (Optional) Switch to a different Atlan domain, if required. If your organization uses a custom domain (for example, _atlan_.mycompany.com ), enter the URL of your Atlan instance and then click Get started . If your organization uses an Atlan domain (for example, _mycompany_.atlan.com ), the Atlan instance URL appears preselected. Click Get started . (Optional) Switch to a different Atlan domain, if required. If your organization uses a custom domain (for example, _atlan_.mycompany.com ), enter the URL of your Atlan instance and then click Get started . After a successful login, the message Updated successfully appears. (Optional) If your data tools are hosted on custom domains, rather than the standard SaaS domain of each tool: Click the Configure custom domain link at the bottom. In the dropdown on the left, select your data tool. In the text box on the right, enter the custom domain you use for that tool. Repeat these steps for each tool hosted on a custom domain. Click the Save button when finished. If your Atlan admin has preconfigured custom domains for data sources , you won't be able to update or remove these selections. Click + Add to configure custom domains for additional data sources as required. Click the Configure custom domain link at the bottom. In the dropdown on the left, select your data tool. In the text box on the right, enter the custom domain you use for that tool. Repeat these steps for each tool hosted on a custom domain. Click the Save button when finished. In the dropdown on the left, select your data tool. In the text box on the right, enter the custom domain you use for that tool. Repeat these steps for each tool hosted on a custom domain. Click the Save button when finished. If your Atlan admin has preconfigured custom domains for data sources , you won't be able to update or remove these selections. Click + Add to configure custom domains for additional data sources as required. You can now close the Options tab. The extension is now ready to use! ð (Optional) Configure custom domains as an admin You need to be an admin user in Atlan to configure custom domains for data sources from the admin center. To configure custom domains, from within Atlan: From the left menu of any screen, click Admin . Under Workspace , click Integrations . Under Apps , expand the Browser extension tile. In the Browser extension tile, for Set up your custom data source. , if your data tools are hosted on custom domains rather than the standard SaaS domain of each tool, click the Configure link to configure them for users in your organization. For Connector , select a supported tool for the browser extension. In the adjacent field, enter the URL of the custom domain for your data source. (Optional) Click + Add to add more. Click Save to save your configuration. info ð ª Did you know? For any supported tools that you have configured, your users won't be able to update or remove these selections. They can, however, add additional custom domains for data sources. For Connector , select a supported tool for the browser extension. In the adjacent field, enter the URL of the custom domain for your data source. (Optional) Click + Add to add more. Click Save to save your configuration. ð ª Did you know? For any supported tools that you have configured, your users won't be able to update or remove these selections. They can, however, add additional custom domains for data sources. (Optional) For Download Atlan extension or share with your team , you can either install the Atlan browser extension for your own use or share the link with your users. Anyone with access to Atlan any admin, member, or guest user and a supported tool can use the browser extension. First, log into Atlan. When using Atlan's browser extension in a supported tool , the extension only reads the URL of your browser tab no other data is accessed. If using Atlan's browser extension on any website , it only reads the favicon, page title, and URL of your browser tab. Learn more about Atlan browser extension security . Access and enrich context in-flow To access context for an asset, from within a supported tool: Log into the supported tool. Open any supported asset. In the lower-right corner of the page, click the small Atlan icon. danger The icon to activate Atlan is not the extension icon that appears at the top of your Chrome browser. This small Atlan icon in the lower right corner of the page is the only way to access the metadata for the asset you are viewing in another tool. The icon to activate Atlan is not the extension icon that appears at the top of your Chrome browser. This small Atlan icon in the lower right corner of the page is the only way to access the metadata for the asset you are viewing in another tool. In the sidebar that appears: Click the tabs and links to view all context about the asset. Make changes to any of the metadata you'd like. Click the tabs and links to view all context about the asset. Make changes to any of the metadata you'd like. Now you can understand and enrich assets without leaving your data tools themselves! ð The Atlan sidebar automatically reloads as you browse your assets in a supported tool to show details about the asset you're currently viewing. Your permissions in Atlan control what metadata you can see and change in the extension. The extension opens a new browser tab on Atlan's discovery page, with the results for that text! ð Add a resource You can link any web page as a resource to your assets in Atlan using the browser extension. To add a web page as a resource to an asset: In the top right of the web page you're viewing, click the Atlan Chrome extension . In the resource clipper menu, under Link this page to an asset , select the asset to which you'd like to add the web page as a resource. Click Save to confirm your selection. (Optional) Once the resource has been linked successfully, click the Open in Atlan button to view the linked asset directly in Atlan. You can now add resources to your assets in Atlan from any website! ð The Tableau extension offers native embeddings directly in your dashboards. See Enable embedded metadata in Tableau for more information. Supported tools Currently, the Atlan browser extension supports assets in the following tools: Amazon QuickSight : analyses, dashboards, and datasets Databricks : databases, schemas, views, and tables dbt Cloud : models and sources in the model editor and dbt docs Google BigQuery : datasets, schemas, views, and tables IBM Cognos Analytics : folders, dashboards, packages, explorations, reports, files, data sources, and modules Looker : dashboards, explores, and folders Microsoft Power BI : dashboards, reports, dataflows, and datasets Mode : collections, reports, queries, and charts Qlik Sense Cloud : apps, datasets, sheets, and spaces Redash : queries, dashboards, and visualizations Salesforce : objects Sigma : datasets, pages, and data elements Snowflake (via Snowsight schema explorer): databases, schemas, tables, views, dynamic tables, streams, and pipes Tableau : dashboards, data sources, workbooks, and metrics. Additionally, you can choose to switch the Tableau extension to offer native embeddings directly in your dashboards. See Enable embedded metadata in Tableau for more information. ThoughtSpot : liveboards, answers, visualizations, and tables MicroStrategy : dossiers, reports, documents Install the extension Configure the extension


Administrators | Atlan Documentation

User management User management is a critical part of data governance. Atlan's user management capabilities should be a mainstay of how you organize and control access for people in your organization. Add and manage users from the admin center It's super simple to invite and remove users from Atlan from the Admin center . You can also manage existing users by adding them to groups, changing their roles, or set up SSO , SCIM , and SMTP configurations. Manage access control from the governance center The Governance center is where you can build access control mechanisms to manage user access . Personas allow you to group users into teams, such as Financial Analysts or Cloud Engineers , and set policies based on the access those personas should have. Purposes are where you can build policies based on the actions or access that a user might need. For example, you can use Atlan's policy-based access controls to manage access to PII and other sensitive data. This is a best practice for data governance. Once you set these policies, Atlan will enforce them throughout your users' experience. This means that users who don't have access to a particular type of data will not be able to see it. Governance workflows help you set up robust controls on data access management, metadata enrichment, new entity creation, and more, with out-of-the-box workflow templates and automated execution. Asset profile The asset profile in Atlan gives you a quick and clear understanding of what a data asset contains. You can think of the asset profile as the TL;DR about your data. The glossary provides key intel on your data assets so you can quickly understand important attributes of your data, such as: Owners of your data, so you know who to ask for clarification. Certificate status, to easily understand if metadata enrichment is still in progress or the asset is ready to be used. Linked assets that are relevant to the term, so you can explore other helpful material. Here are a few of the things that make Atlan's discovery awesome: Intelligent keyword recognition sees through your typos to show exactly what you wanted, no matter what you actually typed. Sort by popularity to quickly discover what assets your teammates are using every day.


Secure Agent | Atlan Documentation

The Atlan Secure Agent is a lightweight, Kubernetes-based application that enables secure metadata extraction. It connects internal systems with Atlan SaaS while keeping sensitive data protected and doesn t require inbound connectivity. Running within an organization s controlled environment, the Secure Agent ensures compliance with security policies and automates metadata processing. Figure 1: The Secure Agent runs in the customer environment and acts as a gateway. Key capabilities The Secure Agent is designed for secure, scalable, and efficient metadata extraction. Security-first architecture Runs entirely within the organization's infrastructure, preventing secrets from leaving its boundary. Uses outbound, encrypted communication to interact with Atlan SaaS. Supports logging and monitoring and integrates with external monitoring systems for auditing and compliance. Scalable metadata extraction A single deployment of the Agent can connect to multiple source systems. Supports multiple concurrent metadata extraction jobs. Uses Kubernetes-based workloads for efficient resource management. Flexible deployment Deploys on cloud-based Kubernetes environments (such as Amazon EKS, Azure AKS, and Google GKE) or on-premises clusters. Scales dynamically based on workload demands. Automated operations Continuously monitors system health and sends heartbeats to Atlan. Captures and uploads execution logs for troubleshooting and auditing. Provides performance insights through metrics and alerts. How it works The Secure Agent follows a job-based execution model where metadata extraction tasks are scheduled and executed within the organization's environment. The workflow typically involves: Atlan triggers a metadata extraction job. The Secure Agent retrieves job details and extracts metadata using source-specific connectors. Extracted metadata is shared with Atlan either through cloud storage or direct ingestion. Atlan workflows process the extracted metadata and publish the assets. Logs and execution status are sent to Atlan for monitoring and auditing. See also Deployment architecture : Learn more about how the Secure Agent integrates with your environment and supports secure metadata extraction. How it works


Playbooks | Atlan Documentation

Get started How to set up playbooks Playbook management How to manage playbooks : Monitor and maintain your playbook workflows. How to automate data profiling : Set up automated data quality checks. Troubleshooting playbooks : Solutions for common playbook issues.


Contracts | Atlan Documentation

Get started Follow these steps to implement contracts in Atlan: Create data contracts Add contract impact analysis in GitHub : Detailed instructions on adding contracts for impact analysis in GitHub.


Integrations | Atlan Documentation

Atlan integrates with a wide range of tools to help you automate workflows, connect with your favorite apps, and manage identity and access. These integrations connect your data catalog with the tools your teams already use, creating a seamless data experience across your tech stack. Key concepts Integration categories : Atlan offers integrations across five categories: project management, communication, collaboration, automation, and identity management. Connection methods : Most integrations use secure authentication methods like OAuth, API keys, or service accounts. Bi-directional sync : Updates flow between Atlan and integrated tools, ensuring data consistency across platforms. Custom webhooks : Extend Atlan's capabilities by building custom integrations using the provided APIs and webhooks. Core offerings Connect with platforms like AWS Lambda to automate data workflows and streamline routine tasks. Integrate with tools like Slack and Microsoft Teams to enhance team collaboration and knowledge sharing. Connect with SMTP for real-time alerts. Integrate with identity providers like Okta and Azure AD for seamless authentication and user management. Connect with tools like Jira and Service Now to link data assets to projects and track data-related tasks. Get started Select an integration Choose from Atlan's available integrations based on your team's tools and workflows. Follow the integration-specific setup guide to establish a secure connection with your tool. Test and activate Verify the integration is working correctly with a test action, then activate for your organization. Need a custom integration? Atlan provides APIs and webhooks that let you build custom integrations with any tool in your tech stack.


Introductory walkthrough - Developer

An introductory walkthrough You might also like our Atlan Platform Essentials certification . Not sure where to start? Allow us to introduce Atlan development through example. 1 We strongly recommend using one of our SDKs to simplify the development process. As a first step, set one up: The SDK is available on Maven Central , ready to be included in your project: Include the latest version of the Java SDK in your project as a dependency. You can also give a specific version instead of the + , if you'd like. The Java SDK uses slf4j for logging purposes. You can include logback as a simple binding mechanism to send any logging information out to your console (standard out). Provide two values to create an Atlan client: Provide your Atlan tenant URL as the first parameter. You can also read the value from an environment variable, if you leave out both parameters. Provide your API token as the second parameter. You can also read the value from another environment variable, by leaving out this parameter. You can then start writing some actual code to run within a static main method. (We'll show some examples of this further below.) Once the block is complete, any resources held by the client (i.e. for caching) will be automatically released. Set up logging for SDK You can also checkout to the advanced configuration section of the SDK to learn about how to set up logging. The SDK is available on PyPI . You can use pip to install it as follows: Provide two values to create an Atlan client: Provide your Atlan tenant URL to the base_url parameter. (You can also do this through environment variables .) Provide your API token to the api_key parameter. (You can also do this through environment variables .) Set up logging for SDK You can also checkout to the advanced configuration section of the SDK to learn about how to set up logging. The SDK is available on Maven Central , ready to be included in your project: Include the latest version of the Java SDK in your project as a dependency. You can also give a specific version instead of the + , if you'd like. The Java SDK uses slf4j for logging purposes. You can include slf4j-simple as a simple binding mechanism to send any logging information out to your console (standard out), along with the kotlin-logging-jvm microutil. Provide two values to create an Atlan client: Provide your Atlan tenant URL as the first parameter. You can also read the value from an environment variable, if you leave out both parameters. Provide your API token as the second parameter. You can also read the value from another environment variable, by leaving out this parameter. You can then start writing some actual code to run within a static main method. (We'll show some examples of this further below.) Once the block is complete, any resources held by the client (i.e. for caching) will be automatically released. Set up logging for SDK You can also checkout to the advanced configuration section of the SDK to learn about how to set up logging. The SDK is available on GitHub , ready to be included in your project: Provide two values to set up connectivity to Atlan: Provide your Atlan tenant URL to the assets.Context() method. If you prefer using the value from an environment variable, you can use assets.NewContext() without any parameters. Provide your API token as the second parameter to the assets.Context() method. (Or again, have it picked up automatically by the assets.NewContext() method.) Set up logging for SDK You can also checkout to the advanced configuration section of the SDK to learn about how to set up logging. Don't forget to give permissions If you want to be able to access existing metadata with an API token, don't forget that you need to assign one or more personas to the API token that grant it access to metadata. Now that you have an SDK installed and configured, you are ready to code! Before we jump straight to code, though, let's first introduce some key concepts in Atlan: What is an asset? In Atlan, we refer to all objects that provide context to your data as assets . Each type of asset in Atlan has a set of: Properties , such as: Certificates Announcements Properties , such as: Relationships to other assets, such as: Schema child tables Table parent schema Table child columns Column parent table Relationships to other assets, such as: Schema child tables Table parent schema Table child columns Column parent table In an object-oriented programming sense, think of an asset as an instance of a class. The structure of an asset (the class itself, in this analogy) is defined by something called a type definition , but that's for another day. So as you can see: There are many different kinds of assets: tables, columns, schemas, databases, business intelligence dashboards, reports, and so on. Assets inter-relate with each other: a table has a parent schema and child columns, a schema has a parent database and child tables, and so on. Different kinds of assets have some common properties (like certificates) and other properties that are unique to that kind of asset (like a columnCount that only exists on tables, not on schemas or databases). When you know the asset When you already know which asset you want to retrieve, you can read it from Atlan using one of its identifiers . We'll discuss these in more detail as part of updates, but for now you can think of them as: is a primary key for an asset: completely unique, but meaningless by itself is a business key for an asset: unique for a given kind of asset, and interpretable You can retrieve an asset using the static get() method on any asset type, providing the client and either the asset's GUID or qualifiedName . (Each asset type is its own unique class in the SDK.) You can retrieve an asset using the asset.get_by_guid() method on the Atlan client, providing both the type of asset you expect to retrieve and its GUID. (Each asset type is its own unique class in the SDK.) You can also retrieve an asset using the asset.get_by_qualified_name() method on the Atlan client, providing the type of asset you expect to retrieve and its qualified_name . (Each asset type is its own unique class in the SDK.) You can retrieve an asset using the static get() method on any asset type, providing the client and either the asset's GUID or qualifiedName . (Each asset type is its own unique class in the SDK.) You can retrieve an asset using the assets.GetByGuid() method on the Atlan client, providing both the type of asset you expect to retrieve and its GUID. (Each asset type is its own unique class in the SDK.) You can also retrieve an asset using the assets.GetByQualifiedName() method on the Atlan client, providing the type of asset you expect to retrieve and its qualifiedName . (Each asset type is its own unique class in the SDK.) Note that the response is strongly typed: If you are retrieving a table, you will get a table back (as long as it exists). You do not need to figure out what properties or relationships exist on a table - the Table class defines them for for you already. In any modern IDE, this means you have type-ahead support for retrieving the properties and relationships from the table variable. You can also refer to the types reference in this portal for full details of every kind of asset. Retrieval by identifier can be more costly than you might expect Even though you are retrieving an asset by an identifier, this can be more costly than you might expect. Retrieving an asset in this way will: Retrieve all its properties and their values Retrieve all its relationships Imagine the asset you are retrieving has 100's or 1000's of these. If you only care about its certificate and any owners, you will be retrieving far more information than you need. When you need to find it first For example, imagine you want to find all tables named MY_TABLE : You can then run the request using Execute() . For example, if you want to know the certificate of the asset you only need to tack that onto the query: Only this line differs from the original query. You can chain as many includeOnResults calls as you want to specify the properties and relationships you want to retrieve for matching assets. Only this line differs from the original query. You can chain as many include_on_results calls as you want to specify the properties and relationships you want to retrieve for matching assets. Only this line differs from the original query. You can chain as many includeOnResults calls as you want to specify the properties and relationships you want to retrieve for matching assets. Only this line differs from the original query. You can include as many attributes in IncludeOnResults as you want to specify the properties and relationships you want to retrieve for matching assets. Also gives the best performance If all you want to do is check or report on metadata, you should have a starting point from the information above. Or, now that you've found an asset of interest, maybe you want to update the asset with additional metadata ? Once again, before we jump to code, let's first understand some key concepts about how Atlan handles updates: Importance of identifiers Most operations on assets are upserts , that is, they could either create (insert) or update a given asset. How do you know which is going to happen? To answer this question, you need to understand how Atlan uniquely identifies each asset. Recall earlier we discussed asset's different identifiers in Atlan . Every asset in Atlan has at least the following two unique identifiers. These are both mandatory for every asset, so no asset can exist without these: Atlan uses globally-unique identifiers (GUIDs) to uniquely identify each asset, globally . They look something like this: As the name implies, GUIDs are: Globally unique (across all systems). Generated in a way that makes it nearly impossible for anything else to ever generate that same ID. 2 Note that this means the GUID itself is not : Meaningful or capable of being interpreted in any way Atlan uses qualifiedName s to uniquely identify assets based on their characteristics. They look something like this: Qualified names are not : Globally unique (across all systems). Instead, they are: Consistently constructed in a meaningful way, making it possible for them to be reconstructed. Note that this means the qualifiedName is: Meaningful and capable of being interpreted How these impact updates Since they are truly unique, operations that include a GUID will only update an asset, not create one. Conversely, operations that take a qualifiedName can: Create an asset, if no exactly-matching qualifiedName is found in Atlan. Update an asset, if an exact-match for the qualifiedName is found in Atlan. These operations also require a typeName , so that if creation does occur the correct type of asset is created. Unintended consequences of this behavior Be careful when using operations with only the qualifiedName . You may end up creating assets when you were only expecting them to be updated or to fail if they did not already exist. This is particularly true when you do not give the exact, case-sensitive qualifiedName of an asset. a/b/c/d is not the same as a/B/c/d when it comes to qualifiedName s. Perhaps this leaves you wondering: why have a qualifiedName at all? The qualifiedName 's purpose is to identify what is a unique asset. Many different tools might all have information about that asset. Having a common "identity" means that many different systems can each independently construct its identifier the same way. If a crawler gets table details from Snowflake it can upsert based on those identity characteristics in Atlan. The crawler will not create duplicate tables every time it runs. This gives idempotency. Looker knows the same identity characteristics for the Snowflake tables and columns. So if you get details from Looker about the tables it uses for reporting, you can link them together in lineage. (Looker can construct the same identifier for the table as Snowflake itself.) These characteristics are not possible using GUIDs alone. Limit to changes only Now that you understand the nuances of identifiers, let's look at how you can update metadata in Atlan. In general, you only need to send changes to Atlan. You do not need to send an entire asset each time you want to make changes to it. For example, imagine you want to mark a table as certified but do not want to change anything else (its name, description, owner details, and so on): You can update an asset without first looking the asset up, if you know (can construct) its identifying qualifiedName . Using the updater() static method on any asset type, you pass in (typically) the qualifiedName and name of the asset. This returns a builder onto which you can then chain any updates. You can then chain onto the returned builder as many updates as you want. In this example, we change the certificate status to VERIFIED . At the end of your chain of updates, you need to build the builder (into an object, in-memory). And then, finally, you need to .save() that object to persist those changes in Atlan (passing the client for the tenant you want to save it in). The response will contain details of the change: whether the asset was created, updated, or nothing happened because the asset already had those changes. You can update an asset without first looking the asset up, if you know (can construct) its identifying qualified_name . Using the updater() class method on any asset type, you pass in (typically) the qualified_name and name of the asset. You can then add onto the returned object as many updates as you want. In this example, we change the certificate status to VERIFIED . And then, finally, you need to client.asset.save() that object to persist those changes in Atlan. The response will contain details of the change: whether the asset was created, updated, or nothing happened because the asset already had those changes. You can update an asset without first looking the asset up, if you know (can construct) its identifying qualifiedName . Using the updater() static method on any asset type, you pass in (typically) the qualifiedName and name of the asset. This returns a builder onto which you can then chain any updates. You can then chain onto the returned builder as many updates as you want. In this example, we change the certificate status to VERIFIED . At the end of your chain of updates, you need to build the builder (into an object, in-memory). And then, finally, you need to .save() that object to persist those changes in Atlan (passing the client for the tenant you want to save it in). The response will contain details of the change: whether the asset was created, updated, or nothing happened because the asset already had those changes. You can update an asset without first looking the asset up, if you know (can construct) its identifying qualifiedName . Using the Updater() method on any asset type, you pass in (typically) the qualifiedName and name of the asset. This returns an object into which you can then place any updates. You can place into the returned object as many updates as you want. In this example, we change the certificate status to VERIFIED . And then, finally, you need to .Save() that object to persist those changes in Atlan. The response will contain details of the change: whether the asset was created, updated, or nothing happened because the asset already had those changes. Atlan will handle idempotency By sending only the changes you want to apply, Atlan can make idempotent updates. Atlan will only attempt to update the asset with the changes you send. Atlan leaves any existing metadata on the asset as-is. If the asset already has the metadata values you are sending, Atlan does nothing. It will not even update audit details like the last update timestamp, and is thus idempotent. What if you want to make changes to many assets, as efficiently as possible? Start by initializing a batch. Through this batch, we can automatically queue up and bulk-upsert assets — in this example, 20 at a time. Be sure to include any details you might need to make a decision about whether to update the asset or not (and what to update it with). You must flush() the batch outside of any loop where you've added assets into it. This ensures any final remaining elements in the batch are still sent to Atlan, even if the batch is not "full". Finally, from the batch you can retrieve the minimal details about any assets it created or updated. Start by initializing a batch. Through this batch, we can automatically queue up and bulk-upsert assets — in this example, 20 at a time. Be sure to include any details you might need to make a decision about whether to update the asset or not (and what to update it with). You can then add() any updated objects directly into the batch you created earlier. The batch itself will handle saving these to Atlan when a sufficient number have been queued up (20, in this example). You must flush() the batch outside of any loop where you've added assets into it. This ensures any final remaining elements in the batch are still sent to Atlan, even if the batch is not "full". Finally, from the batch you can retrieve the minimal details about any assets it created or updated. Start by initializing a batch. Through this batch, we can automatically queue up and bulk-upsert assets — in this example, 20 at a time. Be sure to include any details you might need to make a decision about whether to update the asset or not (and what to update it with). You must flush() the batch outside of any loop where you've added assets into it. This ensures any final remaining elements in the batch are still sent to Atlan, even if the batch is not "full". Finally, from the batch you can retrieve the minimal details about any assets it created or updated. Start by initializing a batch. Through this batch, we can automatically queue up and bulk-upsert assets — in this example, 20 at a time. Be sure to include any details you might need to make a decision about whether to update the asset or not (and what to update it with). You can then add() any updated objects directly into the batch you created earlier. The batch itself will handle saving these to Atlan when a sufficient number have been queued up (20, in this example). You must flush() the batch outside of any loop where you've added assets into it. This ensures any final remaining elements in the batch are still sent to Atlan, even if the batch is not "full". Finally, from the batch you can retrieve the minimal details about any assets it created or updated. Where to go from here Common tasks Common operations on assets, that are available across all assets. Discover actions Common operations on assets, that are available across all assets. Asset-specific Operations that are specific to certain assets. Focus on a specific kind of asset Operations that are specific to certain assets. Focus on a specific kind of asset Governance structures Operations dealing with governance structures, rather than assets. Manage governance structures Operations dealing with governance structures, rather than assets. Manage governance structures Samples Real code samples our customers use to solve particular use cases. Review live samples Real code samples our customers use to solve particular use cases. Review live samples Events Delve deep into the details of the events Atlan triggers. Learn more about events Delve deep into the details of the events Atlan triggers. Learn more about events Note that this is intentionally kept as simple as possible. The walkthrough is not intended to be exhaustive. Where possible, we have cross-referenced other detailed examples elsewhere in the site. ↩ Note that this is intentionally kept as simple as possible. The walkthrough is not intended to be exhaustive. Where possible, we have cross-referenced other detailed examples elsewhere in the site. ↩ There are orders of magnitude lower chances of GUIDs conflicting with each other than there are grains of sand on the planet. (And generating them does not rely on a central ID-assigning registry.) ↩ There are orders of magnitude lower chances of GUIDs conflicting with each other than there are grains of sand on the planet. (And generating them does not rely on a central ID-assigning registry.) ↩


Integration options - Developer

Throughout the portal you can focus on your preferred integration approach (and switch between them as you like): CLI Use the Atlan CLI to manage data contracts for assets in Atlan. Get started with CLI Use the Atlan CLI to manage data contracts for assets in Atlan. Get started with CLI dbt Use dbt's meta field to enrich metadata resources straight from dbt into Atlan. Get started with dbt Use dbt's meta field to enrich metadata resources straight from dbt into Atlan. Get started with dbt Java Pull our Java SDK from Maven Central, just like any other dependency. Get started with Java Pull our Java SDK from Maven Central, just like any other dependency. Get started with Java Python Pull our Python SDK from PyPI, just like any other dependency. Get started with Python Pull our Python SDK from PyPI, just like any other dependency. Get started with Python Kotlin Pull our Java SDK from Maven Central, just like any other dependency. Get started with Kotlin Pull our Java SDK from Maven Central, just like any other dependency. Get started with Kotlin Scala Pull our Java SDK from Maven Central, just like any other dependency. Get started with Scala Pull our Java SDK from Maven Central, just like any other dependency. Get started with Scala Clojure Pull our Java SDK from Maven Central, just like any other dependency. Get started with Clojure Pull our Java SDK from Maven Central, just like any other dependency. Get started with Clojure Go Pull our Go SDK from GitHub, just like any other dependency. Get started with Go Pull our Go SDK from GitHub, just like any other dependency. Get started with Go Events Tap into events Atlan produces to take immediate action, as metadata changes. Get started with events Tap into events Atlan produces to take immediate action, as metadata changes. Get started with events Raw REST API You can call directly into our REST API, though we would recommend the SDKs. Get started with raw REST APIs Raw REST API You can call directly into our REST API, though we would recommend the SDKs. Get started with raw REST APIs