Databricks Offline Store Setup

​

Databricks Workspace Prerequisites

To use Databricks as the offline store for Chalk, Chalk needs access to a Databricks workspace (requires Pricing tier to be Premium or higher, as required functionality (Databricks SQL) is not available on Standard plans) in the same region as your Chalk deployment, along with the following:

1. A Unity Catalog in the same region as your Chalk deployment (must be assigned to the workspace).
2. Create a schema for Chalk within the Unity Catalog, i.e. chalk_offline_store.
3. A Databricks SQL warehouse.
   • The Unity Catalog feature must be enabled.
   • This also requires the Pro feature to be enabled.
4. Recommended warehouse size is 2X-Small (can be scaled up later).

After this phase of setup is complete, Chalk needs the following information:

1. Server hostname (accessible from the SQL warehouse connection details page)
2. HTTP path (accessible from the connection details page)
3. The name of the Unity Catalog.
4. Schema name (within the catalog above)

​

Service Principal

Next, you’ll create a dedicated service principal that Chalk will use to access your Databricks SQL schema.

1. Navigate to Settings > Workspace admin > Identity and access > Service principals
2. Create a service principal for Chalk
3. In Workspace settings > Identity and access > Service principals > Service principal details > Secrets click Generate secret and save the client ID and secret to a secure location.
4. Ensure the service principal has permission to create and drop tables within the schema you created above (Chalk may create a temporary table to cache data that can be accessed across multiple connections and then drop the table when it’s no longer used). It’s a best practice to only grant ownership access to this schema. It’s also recommended to ensure this user cannot create new users / modify permissions. There are some optional Chalk features that can be used to query other tables in the same database (catalog) as your offline store: https://docs.chalk.ai/docs/changelog#spine-sql-query. If these are to be used it’s recommended to grant read-only access to the other schemas.

grant use catalog on catalog <CATALOG HERE> to `<SERVICE PRINCIPAL UUID HERE>;
grant all privileges on schema <CATALOG HERE>.<SCHEMA HERE> to `<SERVICE PRINCIPAL UUID HERE>`;

It’s worth noting that to grant the service principal access to the SQL warehouse above it was necessary to use the Databricks SQL CLI and give the service principal the CAN_MANAGE permission. The service_principal_name actually corresponds to the service principal’s unique ID, not the name displayed in the console (these names are not necessarily unique and the permissions cannot be ambiguous). This apparently can be done from the web portal on AWS but not GCP.

​

Staging Storage Bucket

You will need to set up a staging location that Chalk will use for data uploaded to the offline store in Databricks.

1. Create a dedicated object storage bucket, i.e. S3, Google Cloud Storage, etc.
2. Navigate to Settings > Shared Resources > Background Persistence in your Chalk environment and click the Edit JSON button
3. Replace the bq_upload_bucket field with your storage bucket URI and save and apply changes
4. The offline writer managed identity will also need exclusive access to the storage bucket

​

Storage Credential and External Location

Next, you’ll create a storage credential and external location that Chalk will use to access your staging bucket.

1. Click Catalog in the sidebar.
2. Click the + in the Catalog sidebar.
3. Click Add a storage credential and follow the instructions (the storage credential should probably be in the same cloud account as your Databricks workspace AND your bucket). You’ll need to give the underlying role / service account etc. access to the bucket.
4. Click Add an external location and specify the storage credential and the bucket you created previously.
5. Grant the service principal all privileges on the external location via SQL.
6. Recommended: Run a LIST command on your external location using the service account.

grant usage on external location `<EXTERNAL LOCATION HERE>` to `<SERVICE PRINCIPAL UUID HERE>`;
-- You can run this using the service principal to ensure proper access.
-- n.b. this requires single-quotes (not backticks) and
list '<chalk cloud storage url>';