Databricks

Connect Erathos to Databricks.

You can connect the Erathos platform to any Databricks instance, regardless of the cloud environment it is hosted in (AWS, Azure, or Google Cloud).

To connect to your Databricks SQL warehouse the following information is necessary: Server Hostname, HTTP path and Catalog Name. Furthermore, you must choose either Personal Access Token (PAT) or OAuth as an authentication method.

The tutorial below shows you how to obtain these credentials.

Databricks Setup

Create a SQL Warehouse (Optional)

The first step is to create a new SQL Warehouse. If you already have an existing one, you can skip this step.

In the sidebar menu select SQL Warehouse and then click on the Create SQL Warehouse on the right side of the screen. A quick configuration window will be shown.

Name your SQL warehouse and adjust the hardware settings according to your usage needs (i.e., cluster size, auto stop, scaling). When the configuration is done, click on Create.

Basic Credentials

Select the desired SQL Warehouse and navigate to the Connection details tab to obtain the Server Hostname and HTTP path credentials.

Identify the Catalog Name by selecting Catalog in the left sidebar and checking the list of available catalogs at the SQL Warehouse. Choose the name of one of the available options or create a new catalog.

After clicking on a catalog option, navigate to Permissions tab and confirm that the appropriate user credentials have Data editor privileges in the catalog (the owner of the access token or the service principal for OAuth method).

Personal Access Token

In the Connection Details tab for the chosen SQL Warehouse, select the Create a personal access token option to create a new Access Token:

Click the Generate new token button, name the new token for identification purposes, and specify how long before the token should expire.

Warning: the token value will not be exhibited again, so confirm you have copied and pasted it somewhere safe.

OAuth

Navigate to the Databricks accounts initial page and click on User Management in the sidebar. Then, select the Service Principals tab and click on the Add service principal button.

In order to facilitate identifying the credential used for Erathos connection name it appropriately. Click on Add service principal to complete the creation process.

Returning to the list of available service principals, click on your newly created one. Head to the Credentials & secrets tab.

Click on the Generate secret button at the bottom of the page to create a new pair of values for Client ID / Client Secret.

Warning: the secret value will not be exhibited again, so confirm you have copied and pasted it somewhere safe.

Next, you will need to confirm the workspace's access permissions for the new service principal. To do so, select Workspaces in the sidebar and click on the chosen workspace.

Navigate to Permissions tab and click on Add permissions to open a dialog window.

Search the service principal by typing the name you chose earlier and complete the process by selecting the level of access attributed to the principal credentials.

Connect to Erathos

To connect Erathos to your Databricks:

1. Create a New Destination

   • Go to Settings > Destination.

   • Select Databricks.

2. Fill Out the Form

   • Set the Server Hostname, HTTP path and Catalog Name.

   • Choose between Personal Access Token (PAT) or OAuth authentication method:

     • (PAT) Paste the user Access Token value.

     • (OAuth) Paste the service principal's Client ID and Client Secret.

3. Complete the Setup

If you need help connecting to your Databricks, reach out to our team at [email protected].