GCP Cloud Deployment

​

Background

Chalk enables innovative machine learning teams to focus on building the unique products and models that make their business stand out. Behind the scenes Chalk seamlessly handles data infrastructure with a best-in-class developer experience.

Chalk is both a framework and a platform — developers can write code using familiar Python packages, and deploy that code to Chalk’s platform. In the Customer Cloud deployment, Chalk runs & administers its platform on the customer’s cloud account. Chalk’s managed infrastructure then executes-customer defined pipelines to compute “feature” data for machine learning applications. Chalk then serves this data back to customer applications for online inference and to customer data teams for training set generation.

​

Architecture Diagram

1. 1Creating secrets:The API server can be configured to have write-only access to the secret store.
2. 2Reading secrets:Secret access can be restricted entirely to the data plane.
3. 3Online store:Chalk supports several online feature stores, which are used for caching feature values. On GCP, Chalk supports Bigtable, Cloud Memorystore, and PostgreSQL.

​

IAM Requirements

Chalk requires certain IAM permissions to manage infrastructure in your cloud account. Typically, Chalk recommends:

• Creating a new project in your GCP organization
• Creating a new role in your root GCP organization with the permissions in this document
• Granting Chalk’s customer-scoped management service account the custom role

​

Setup permissions

Chalk recommends creating a custom “role” in your organization with the following permissions granted. At a high level, Chalk needs the ability to provision the key components of your infrastructure:

• Storage resources (buckets for dataset storage and bulk insertion into offline storage)
• Artifact Registry resources (repositories for Chalk deployment images)
• Networking resources (LBs, VPCs, DNS zones, etc.)
• IAM resources (e.g. creating service accounts for workload identity, etc.)
• Kubernetes resources (GKE)
• Online storage (CloudSQL, or Memorystore, or Bigtable)
• Offline storage (typically BigQuery or Snowflake)

artifactregistry.repositories.create
artifactregistry.repositories.get
artifactregistry.repositories.getIamPolicy
artifactregistry.repositories.setIamPolicy
cloudbuild.builds.create
cloudbuild.builds.get
cloudbuild.builds.list
cloudbuild.builds.update
cloudbuild.integrations.get
cloudbuild.integrations.list
cloudnotifications.activities.list
cloudscheduler.jobs.create
cloudscheduler.jobs.get
cloudscheduler.locations.list
cloudsql.backupRuns.list
cloudsql.databases.create
cloudsql.databases.delete
cloudsql.databases.get
cloudsql.databases.list
cloudsql.databases.update
cloudsql.instances.clone
cloudsql.instances.create
cloudsql.instances.delete
cloudsql.instances.export
cloudsql.instances.failover
cloudsql.instances.get
cloudsql.instances.import
cloudsql.instances.list
cloudsql.instances.login
cloudsql.instances.promoteReplica
cloudsql.instances.restart
cloudsql.instances.startReplica
cloudsql.instances.stopReplica
cloudsql.instances.update
cloudsql.sslCerts.create
cloudsql.sslCerts.delete
cloudsql.sslCerts.get
cloudsql.sslCerts.list
cloudsql.users.create
cloudsql.users.delete
cloudsql.users.get
cloudsql.users.list
cloudsql.users.update
compute.acceleratorTypes.list
compute.addresses.create
compute.addresses.delete
compute.addresses.deleteInternal
compute.addresses.get
compute.addresses.list
compute.autoscalers.list
compute.autoscalers.update
compute.backendBuckets.create
compute.backendBuckets.list
compute.backendServices.create
compute.backendServices.delete
compute.backendServices.get
compute.backendServices.list
compute.backendServices.update
compute.backendServices.use
compute.diskTypes.list
compute.disks.create
compute.disks.delete
compute.disks.get
compute.disks.list
compute.disks.setLabels
compute.disks.useReadOnly
compute.firewallPolicies.create
compute.firewallPolicies.delete
compute.firewallPolicies.get
compute.firewallPolicies.getIamPolicy
compute.firewallPolicies.list
compute.firewallPolicies.setIamPolicy
compute.firewallPolicies.update
compute.firewallPolicies.use
compute.firewalls.create
compute.firewalls.delete
compute.firewalls.get
compute.firewalls.list
compute.firewalls.update
compute.forwardingRules.create
compute.forwardingRules.delete
compute.forwardingRules.get
compute.forwardingRules.list
compute.globalAddresses.create
compute.globalAddresses.createInternal
compute.globalAddresses.get
compute.globalAddresses.list
compute.globalAddresses.setLabels
compute.globalForwardingRules.get
compute.globalForwardingRules.list
compute.globalNetworkEndpointGroups.attachNetworkEndpoints
compute.globalNetworkEndpointGroups.create
compute.globalNetworkEndpointGroups.delete
compute.globalNetworkEndpointGroups.detachNetworkEndpoints
compute.globalNetworkEndpointGroups.get
compute.globalNetworkEndpointGroups.list
compute.globalNetworkEndpointGroups.use
compute.globalOperations.get
compute.healthChecks.create
compute.healthChecks.get
compute.healthChecks.list
compute.healthChecks.useReadOnly
compute.httpHealthChecks.list
compute.httpsHealthChecks.list
compute.images.create
compute.images.get
compute.images.list
compute.images.useReadOnly
compute.instanceGroupManagers.create
compute.instanceGroupManagers.get
compute.instanceGroupManagers.list
compute.instanceGroupManagers.update
compute.instanceGroups.create
compute.instanceGroups.delete
compute.instanceGroups.get
compute.instanceGroups.list
compute.instanceGroups.update
compute.instanceTemplates.create
compute.instanceTemplates.get
compute.instanceTemplates.list
compute.instanceTemplates.useReadOnly
compute.instances.addAccessConfig
compute.instances.addResourcePolicies
compute.instances.attachDisk
compute.instances.create
compute.instances.delete
compute.instances.deleteAccessConfig
compute.instances.detachDisk
compute.instances.get
compute.instances.list
compute.instances.listEffectiveTags
compute.instances.listReferrers
compute.instances.osAdminLogin
compute.instances.osLogin
compute.instances.reset
compute.instances.resume
compute.instances.setDeletionProtection
compute.instances.setDiskAutoDelete
compute.instances.setLabels
compute.instances.setMachineResources
compute.instances.setMachineType
compute.instances.setMetadata
compute.instances.setMinCpuPlatform
compute.instances.setScheduling
compute.instances.setServiceAccount
compute.instances.setTags
compute.instances.start
compute.instances.stop
compute.instances.suspend
compute.instances.updateAccessConfig
compute.instances.updateDisplayDevice
compute.instances.updateNetworkInterface
compute.instances.updateShieldedInstanceConfig
compute.instances.use
compute.interconnectAttachments.list
compute.machineImages.create
compute.machineTypes.list
compute.networkEndpointGroups.create
compute.networkEndpointGroups.delete
compute.networkEndpointGroups.get
compute.networkEndpointGroups.list
compute.networkEndpointGroups.use
compute.networks.addPeering
compute.networks.create
compute.networks.delete
compute.networks.get
compute.networks.getEffectiveFirewalls
compute.networks.list
compute.networks.removePeering
compute.networks.switchToCustomMode
compute.networks.update
compute.networks.updatePeering
compute.networks.updatePolicy
compute.networks.use
compute.projects.get
compute.projects.setCommonInstanceMetadata
compute.regionBackendServices.create
compute.regionBackendServices.delete
compute.regionBackendServices.get
compute.regionBackendServices.list
compute.regionFirewallPolicies.create
compute.regionFirewallPolicies.list
compute.regionHealthChecks.create
compute.regionHealthChecks.list
compute.regionNetworkEndpointGroups.create
compute.regionNetworkEndpointGroups.get
compute.regionNetworkEndpointGroups.use
compute.regionOperations.get
compute.regions.list
compute.resourcePolicies.create
compute.resourcePolicies.delete
compute.resourcePolicies.get
compute.resourcePolicies.list
compute.resourcePolicies.use
compute.routers.create
compute.routers.delete
compute.routers.get
compute.routers.list
compute.routers.list
compute.routers.update
compute.routers.use
compute.routes.create
compute.routes.delete
compute.routes.list
compute.securityPolicies.create
compute.securityPolicies.list
compute.snapshots.list
compute.sslCertificates.create
compute.sslCertificates.delete
compute.sslCertificates.get
compute.sslCertificates.list
compute.sslPolicies.create
compute.sslPolicies.get
compute.sslPolicies.list
compute.subnetworks.create
compute.subnetworks.delete
compute.subnetworks.get
compute.subnetworks.list
compute.subnetworks.use
compute.subnetworks.useExternalIp
compute.targetHttpProxies.get
compute.targetHttpProxies.list
compute.targetHttpsProxies.get
compute.targetHttpsProxies.list
compute.targetPools.create
compute.targetPools.delete
compute.targetPools.get
compute.targetPools.list
compute.targetPools.update
compute.targetSslProxies.list
compute.targetTcpProxies.list
compute.urlMaps.create
compute.urlMaps.delete
compute.urlMaps.get
compute.urlMaps.list
compute.urlMaps.update
compute.vpnTunnels.list
compute.zoneOperations.get
compute.zones.list
container.apiServices.create
container.apiServices.delete
container.apiServices.get
container.apiServices.getStatus
container.apiServices.list
container.apiServices.update
container.apiServices.updateStatus
container.auditSinks.create
container.auditSinks.delete
container.auditSinks.get
container.auditSinks.list
container.auditSinks.update
container.backendConfigs.create
container.backendConfigs.delete
container.backendConfigs.get
container.backendConfigs.list
container.backendConfigs.update
container.bindings.create
container.bindings.delete
container.bindings.get
container.bindings.list
container.bindings.update
container.certificateSigningRequests.approve
container.certificateSigningRequests.create
container.certificateSigningRequests.delete
container.certificateSigningRequests.get
container.certificateSigningRequests.getStatus
container.certificateSigningRequests.list
container.certificateSigningRequests.update
container.certificateSigningRequests.updateStatus
container.clusterRoleBindings.create
container.clusterRoleBindings.delete
container.clusterRoleBindings.get
container.clusterRoleBindings.list
container.clusterRoleBindings.update
container.clusterRoles.bind
container.clusterRoles.create
container.clusterRoles.delete
container.clusterRoles.escalate
container.clusterRoles.get
container.clusterRoles.list
container.clusterRoles.update
container.clusters.create
container.clusters.createTagBinding
container.clusters.delete
container.clusters.deleteTagBinding
container.clusters.get
container.clusters.getCredentials
container.clusters.impersonate
container.clusters.list
container.clusters.listEffectiveTags
container.clusters.listTagBindings
container.clusters.update
container.componentStatuses.get
container.componentStatuses.list
container.configMaps.create
container.configMaps.delete
container.configMaps.get
container.configMaps.list
container.configMaps.update
container.controllerRevisions.create
container.controllerRevisions.delete
container.controllerRevisions.get
container.controllerRevisions.list
container.controllerRevisions.update
container.cronJobs.create
container.cronJobs.delete
container.cronJobs.get
container.cronJobs.getStatus
container.cronJobs.list
container.cronJobs.update
container.cronJobs.updateStatus
container.csiDrivers.create
container.csiDrivers.delete
container.csiDrivers.get
container.csiDrivers.list
container.csiDrivers.update
container.csiNodeInfos.create
container.csiNodeInfos.delete
container.csiNodeInfos.get
container.csiNodeInfos.list
container.csiNodeInfos.update
container.csiNodes.create
container.csiNodes.delete
container.csiNodes.get
container.csiNodes.list
container.csiNodes.update
container.customResourceDefinitions.create
container.customResourceDefinitions.delete
container.customResourceDefinitions.get
container.customResourceDefinitions.getStatus
container.customResourceDefinitions.list
container.customResourceDefinitions.update
container.customResourceDefinitions.updateStatus
container.daemonSets.create
container.daemonSets.delete
container.daemonSets.get
container.daemonSets.getStatus
container.daemonSets.list
container.daemonSets.update
container.daemonSets.updateStatus
container.deployments.create
container.deployments.delete
container.deployments.get
container.deployments.getScale
container.deployments.getStatus
container.deployments.list
container.deployments.rollback
container.deployments.update
container.deployments.updateScale
container.deployments.updateStatus
container.endpointSlices.create
container.endpointSlices.delete
container.endpointSlices.get
container.endpointSlices.list
container.endpointSlices.update
container.endpoints.create
container.endpoints.delete
container.endpoints.get
container.endpoints.list
container.endpoints.update
container.events.create
container.events.delete
container.events.get
container.events.list
container.events.update
container.frontendConfigs.create
container.frontendConfigs.delete
container.frontendConfigs.get
container.frontendConfigs.list
container.frontendConfigs.update
container.horizontalPodAutoscalers.create
container.horizontalPodAutoscalers.delete
container.horizontalPodAutoscalers.get
container.horizontalPodAutoscalers.getStatus
container.horizontalPodAutoscalers.list
container.horizontalPodAutoscalers.update
container.horizontalPodAutoscalers.updateStatus
container.hostServiceAgent.use
container.ingresses.create
container.ingresses.delete
container.ingresses.get
container.ingresses.getStatus
container.ingresses.list
container.ingresses.update
container.ingresses.updateStatus
container.initializerConfigurations.create
container.initializerConfigurations.delete
container.initializerConfigurations.get
container.initializerConfigurations.list
container.initializerConfigurations.update
container.jobs.create
container.jobs.delete
container.jobs.get
container.jobs.getStatus
container.jobs.list
container.jobs.update
container.jobs.updateStatus
container.leases.create
container.leases.delete
container.leases.get
container.leases.list
container.leases.update
container.limitRanges.create
container.limitRanges.delete
container.limitRanges.get
container.limitRanges.list
container.limitRanges.update
container.localSubjectAccessReviews.create
container.localSubjectAccessReviews.list
container.managedCertificates.create
container.managedCertificates.delete
container.managedCertificates.get
container.managedCertificates.list
container.managedCertificates.update
container.mutatingWebhookConfigurations.create
container.mutatingWebhookConfigurations.delete
container.mutatingWebhookConfigurations.get
container.mutatingWebhookConfigurations.list
container.mutatingWebhookConfigurations.update
container.namespaces.create
container.namespaces.delete
container.namespaces.finalize
container.namespaces.get
container.namespaces.getStatus
container.namespaces.list
container.namespaces.update
container.namespaces.updateStatus
container.networkPolicies.create
container.networkPolicies.delete
container.networkPolicies.get
container.networkPolicies.list
container.networkPolicies.update
container.nodes.create
container.nodes.delete
container.nodes.get
container.nodes.getStatus
container.nodes.list
container.nodes.proxy
container.nodes.update
container.nodes.updateStatus
container.operations.get
container.operations.list
container.persistentVolumeClaims.create
container.persistentVolumeClaims.delete
container.persistentVolumeClaims.get
container.persistentVolumeClaims.getStatus
container.persistentVolumeClaims.list
container.persistentVolumeClaims.update
container.persistentVolumeClaims.updateStatus
container.persistentVolumes.create
container.persistentVolumes.delete
container.persistentVolumes.get
container.persistentVolumes.getStatus
container.persistentVolumes.list
container.persistentVolumes.update
container.persistentVolumes.updateStatus
container.petSets.create
container.petSets.delete
container.petSets.get
container.petSets.list
container.petSets.update
container.petSets.updateStatus
container.podDisruptionBudgets.create
container.podDisruptionBudgets.delete
container.podDisruptionBudgets.get
container.podDisruptionBudgets.getStatus
container.podDisruptionBudgets.list
container.podDisruptionBudgets.update
container.podDisruptionBudgets.updateStatus
container.podPresets.create
container.podPresets.delete
container.podPresets.get
container.podPresets.list
container.podPresets.update
container.podSecurityPolicies.create
container.podSecurityPolicies.delete
container.podSecurityPolicies.get
container.podSecurityPolicies.list
container.podSecurityPolicies.update
container.podSecurityPolicies.use
container.podTemplates.create
container.podTemplates.delete
container.podTemplates.get
container.podTemplates.list
container.podTemplates.update
container.pods.attach
container.pods.create
container.pods.delete
container.pods.evict
container.pods.exec
container.pods.get
container.pods.getLogs
container.pods.getStatus
container.pods.initialize
container.pods.list
container.pods.portForward
container.pods.proxy
container.pods.update
container.pods.updateStatus
container.priorityClasses.create
container.priorityClasses.delete
container.priorityClasses.get
container.priorityClasses.list
container.priorityClasses.update
container.replicaSets.create
container.replicaSets.delete
container.replicaSets.get
container.replicaSets.getScale
container.replicaSets.getStatus
container.replicaSets.list
container.replicaSets.update
container.replicaSets.updateScale
container.replicaSets.updateStatus
container.replicationControllers.create
container.replicationControllers.delete
container.replicationControllers.get
container.replicationControllers.getScale
container.replicationControllers.getStatus
container.replicationControllers.list
container.replicationControllers.update
container.replicationControllers.updateScale
container.replicationControllers.updateStatus
container.resourceQuotas.create
container.resourceQuotas.delete
container.resourceQuotas.get
container.resourceQuotas.getStatus
container.resourceQuotas.list
container.resourceQuotas.update
container.resourceQuotas.updateStatus
container.roleBindings.create
container.roleBindings.delete
container.roleBindings.get
container.roleBindings.list
container.roleBindings.update
container.roles.bind
container.roles.create
container.roles.delete
container.roles.escalate
container.roles.get
container.roles.list
container.roles.update
container.runtimeClasses.create
container.runtimeClasses.delete
container.runtimeClasses.get
container.runtimeClasses.list
container.runtimeClasses.update
container.scheduledJobs.create
container.scheduledJobs.delete
container.scheduledJobs.get
container.scheduledJobs.list
container.scheduledJobs.update
container.scheduledJobs.updateStatus
container.secrets.create
container.secrets.delete
container.secrets.get
container.secrets.list
container.secrets.update
container.selfSubjectAccessReviews.create
container.selfSubjectAccessReviews.list
container.selfSubjectRulesReviews.create
container.serviceAccounts.create
container.serviceAccounts.createToken
container.serviceAccounts.delete
container.serviceAccounts.get
container.serviceAccounts.list
container.serviceAccounts.update
container.services.create
container.services.delete
container.services.get
container.services.getStatus
container.services.list
container.services.proxy
container.services.update
container.services.updateStatus
container.statefulSets.create
container.statefulSets.delete
container.statefulSets.get
container.statefulSets.getScale
container.statefulSets.getStatus
container.statefulSets.list
container.statefulSets.update
container.statefulSets.updateScale
container.statefulSets.updateStatus
container.storageClasses.create
container.storageClasses.delete
container.storageClasses.get
container.storageClasses.list
container.storageClasses.update
container.storageStates.create
container.storageStates.delete
container.storageStates.get
container.storageStates.getStatus
container.storageStates.list
container.storageStates.update
container.storageStates.updateStatus
container.storageVersionMigrations.create
container.storageVersionMigrations.delete
container.storageVersionMigrations.get
container.storageVersionMigrations.getStatus
container.storageVersionMigrations.list
container.storageVersionMigrations.update
container.storageVersionMigrations.updateStatus
container.subjectAccessReviews.create
container.subjectAccessReviews.list
container.thirdPartyObjects.create
container.thirdPartyObjects.delete
container.thirdPartyObjects.get
container.thirdPartyObjects.list
container.thirdPartyObjects.update
container.thirdPartyResources.create
container.thirdPartyResources.delete
container.thirdPartyResources.get
container.thirdPartyResources.list
container.thirdPartyResources.update
container.tokenReviews.create
container.updateInfos.create
container.updateInfos.delete
container.updateInfos.get
container.updateInfos.list
container.updateInfos.update
container.validatingWebhookConfigurations.create
container.validatingWebhookConfigurations.delete
container.validatingWebhookConfigurations.get
container.validatingWebhookConfigurations.list
container.validatingWebhookConfigurations.update
container.volumeAttachments.create
container.volumeAttachments.delete
container.volumeAttachments.get
container.volumeAttachments.getStatus
container.volumeAttachments.list
container.volumeAttachments.update
container.volumeAttachments.updateStatus
container.volumeSnapshotClasses.create
container.volumeSnapshotClasses.delete
container.volumeSnapshotClasses.get
container.volumeSnapshotClasses.list
container.volumeSnapshotClasses.update
container.volumeSnapshotContents.create
container.volumeSnapshotContents.delete
container.volumeSnapshotContents.get
container.volumeSnapshotContents.getStatus
container.volumeSnapshotContents.list
container.volumeSnapshotContents.update
container.volumeSnapshotContents.updateStatus
container.volumeSnapshots.create
container.volumeSnapshots.delete
container.volumeSnapshots.get
container.volumeSnapshots.getStatus
container.volumeSnapshots.list
container.volumeSnapshots.update
container.volumeSnapshots.updateStatus
dns.managedZones.create
dns.managedZones.get
eventarc.triggers.get
iam.roles.create
iam.roles.delete
iam.roles.get
iam.roles.list
iam.roles.undelete
iam.roles.update
iam.serviceAccountKeys.create
iam.serviceAccountKeys.delete
iam.serviceAccountKeys.get
iam.serviceAccountKeys.list
iam.serviceAccounts.actAs
iam.serviceAccounts.create
iam.serviceAccounts.delete
iam.serviceAccounts.disable
iam.serviceAccounts.enable
iam.serviceAccounts.get
iam.serviceAccounts.getAccessToken
iam.serviceAccounts.getIamPolicy
iam.serviceAccounts.list
iam.serviceAccounts.setIamPolicy
iam.serviceAccounts.update
logging.buckets.create
logging.buckets.delete
logging.buckets.get
logging.buckets.list
logging.buckets.undelete
logging.buckets.update
logging.links.create
logging.links.delete
logging.links.get
logging.links.list
logging.locations.list
logging.logEntries.create
logging.logEntries.download
logging.logEntries.list
logging.logServiceIndexes.list
logging.logServices.list
logging.operations.list
logging.privateLogEntries.list
logging.queries.create
logging.queries.delete
logging.queries.get
logging.queries.list
logging.queries.listShared
logging.queries.share
logging.queries.update
logging.queries.updateShared
logging.sinks.create
logging.sinks.delete
logging.sinks.get
logging.sinks.list
logging.sinks.update
logging.views.access
logging.views.get
logging.views.list
memorystore.backupCollections.get
memorystore.backupCollections.list
memorystore.backups.delete 
memorystore.backups.export
memorystore.backups.get
memorystore.backups.list 
memorystore.instances.backup
memorystore.instances.connect 
memorystore.instances.create
memorystore.instances.delete
memorystore.instances.get
memorystore.instances.list 
memorystore.instances.update
memorystore.locations.get
memorystore.locations.list 
memorystore.operations.cancel
memorystore.operations.delete
memorystore.operations.get
memorystore.operations.list 
monitoring.alertPolicies.create
monitoring.alertPolicies.delete
monitoring.alertPolicies.get
monitoring.alertPolicies.list
monitoring.alertPolicies.update
monitoring.dashboards.create
monitoring.dashboards.get
monitoring.dashboards.list
monitoring.dashboards.update
monitoring.groups.list
monitoring.metricDescriptors.list
monitoring.monitoredResourceDescriptors.get
monitoring.monitoredResourceDescriptors.list
monitoring.notificationChannelDescriptors.list
monitoring.notificationChannels.list
monitoring.services.list
monitoring.slos.list
monitoring.timeSeries.list
monitoring.uptimeCheckConfigs.create
monitoring.uptimeCheckConfigs.list
networkconnectivity.serviceConnectionPolicies.create
networkconnectivity.serviceConnectionPolicies.delete
networkconnectivity.serviceConnectionPolicies.get
networkconnectivity.serviceConnectionPolicies.list
networkconnectivity.serviceConnectionPolicies.update
networkconnectivity.operations.cancel
networkconnectivity.operations.delete
networkconnectivity.operations.get
networkconnectivity.operations.list
opsconfigmonitoring.resourceMetadata.list
orgpolicy.policy.get
pubsub.snapshots.create
pubsub.snapshots.list
pubsub.subscriptions.consume
pubsub.subscriptions.create
pubsub.subscriptions.delete
pubsub.subscriptions.get
pubsub.subscriptions.getIamPolicy
pubsub.subscriptions.list
pubsub.subscriptions.setIamPolicy
pubsub.subscriptions.update
pubsub.topics.attachSubscription
pubsub.topics.create
pubsub.topics.delete
pubsub.topics.detachSubscription
pubsub.topics.get
pubsub.topics.getIamPolicy
pubsub.topics.list
pubsub.topics.publish
pubsub.topics.setIamPolicy
pubsub.topics.update
recommender.cloudsqlIdleInstanceRecommendations.list
recommender.cloudsqlInstanceOutOfDiskRecommendations.list
recommender.cloudsqlOverprovisionedInstanceRecommendations.list
recommender.containerDiagnosisInsights.get
recommender.containerDiagnosisInsights.list
recommender.containerDiagnosisRecommendations.get
recommender.containerDiagnosisRecommendations.list
recommender.containerDiagnosisRecommendations.update
recommender.runServiceIdentityRecommendations.list
recommender.runServiceIdentityRecommendations.update
redis.instances.create
redis.instances.delete
redis.instances.export
redis.instances.get
redis.instances.getAuthString
redis.instances.import
redis.instances.list
redis.instances.rescheduleMaintenance
redis.instances.update
redis.instances.updateAuth
redis.instances.upgrade
redis.operations.get
redis.operations.list
resourcemanager.projects.get
resourcemanager.projects.list
resourcemanager.projects.getIamPolicy
resourcemanager.projects.setIamPolicy
run.executions.get
run.jobs.create
run.jobs.delete
run.jobs.list
run.locations.list
run.revisions.delete
run.revisions.get
run.revisions.list
run.routes.list
run.services.create
run.services.delete
run.services.get
run.services.getIamPolicy
run.services.list
run.services.listEffectiveTags
run.services.listTagBindings
run.services.setIamPolicy
run.services.update
secretmanager.secrets.create
secretmanager.secrets.delete
secretmanager.secrets.get
secretmanager.secrets.getIamPolicy
secretmanager.secrets.list
secretmanager.secrets.setIamPolicy
secretmanager.secrets.update
secretmanager.versions.access
secretmanager.versions.add
secretmanager.versions.destroy
secretmanager.versions.disable
secretmanager.versions.enable
secretmanager.versions.get
secretmanager.versions.list
servicenetworking.services.get
servicenetworking.operations.get
servicenetworking.services.deleteConnection
servicenetworking.services.addPeering
serviceusage.quotas.get
serviceusage.services.enable
serviceusage.services.get
serviceusage.services.list
stackdriver.projects.get
storage.buckets.create
storage.buckets.delete
storage.buckets.get
storage.buckets.getIamPolicy
storage.buckets.list
storage.buckets.listEffectiveTags
storage.buckets.setIamPolicy
storage.buckets.update
storage.objects.create
storage.objects.delete
storage.objects.get
storage.objects.getIamPolicy
storage.objects.list
storage.objects.setIamPolicy
vpcaccess.connectors.create
vpcaccess.connectors.delete
vpcaccess.connectors.get
vpcaccess.connectors.list
vpcaccess.locations.list
vpcaccess.operations.get

​

BigQuery Offline Storage

For BigQuery offline storage, Chalk requires the following provisioning permissions:

bigquery.config.update
bigquery.config.get
bigquery.connections.create
bigquery.connections.delete
bigquery.connections.get
bigquery.connections.list
bigquery.connections.update
bigquery.connections.updateTag
bigquery.connections.use
bigquery.connections.delegate
bigquery.dataPolicies.create
bigquery.dataPolicies.delete
bigquery.dataPolicies.get
bigquery.dataPolicies.getIamPolicy
bigquery.dataPolicies.list
bigquery.dataPolicies.maskedGet
bigquery.dataPolicies.setIamPolicy
bigquery.dataPolicies.update
bigquery.datasets.create
bigquery.datasets.createTagBinding
bigquery.datasets.delete
bigquery.datasets.deleteTagBinding
bigquery.datasets.get
bigquery.datasets.getIamPolicy
bigquery.datasets.link
bigquery.datasets.listTagBindings
bigquery.datasets.setIamPolicy
bigquery.datasets.update
bigquery.datasets.updateTag
bigquery.jobs.create
bigquery.jobs.get
bigquery.jobs.list
bigquery.jobs.listAll
bigquery.jobs.listExecutionMetadata
bigquery.jobs.delete
bigquery.jobs.update
bigquery.models.create
bigquery.models.delete
bigquery.models.getData
bigquery.models.getMetadata
bigquery.models.list
bigquery.models.updateData
bigquery.models.updateMetadata
bigquery.models.export
bigquery.models.updateTag
bigquery.readsessions.create
bigquery.readsessions.getData
bigquery.readsessions.update
bigquery.rowAccessPolicies.create
bigquery.rowAccessPolicies.delete
bigquery.rowAccessPolicies.getFilteredData
bigquery.rowAccessPolicies.list
bigquery.rowAccessPolicies.getIamPolicy
bigquery.rowAccessPolicies.setIamPolicy
bigquery.rowAccessPolicies.update
bigquery.routines.create
bigquery.routines.delete
bigquery.routines.get
bigquery.routines.list
bigquery.routines.update
bigquery.routines.updateTag
bigquery.savedqueries.create
bigquery.savedqueries.delete
bigquery.savedqueries.get
bigquery.savedqueries.list
bigquery.savedqueries.update
bigquery.tables.create
bigquery.tables.createIndex
bigquery.tables.createSnapshot
bigquery.tables.delete
bigquery.tables.deleteIndex
bigquery.tables.deleteSnapshot
bigquery.tables.export
bigquery.tables.get
bigquery.tables.getData
bigquery.tables.getIamPolicy
bigquery.tables.list
bigquery.tables.replicateData
bigquery.tables.restoreSnapshot
bigquery.tables.setCategory
bigquery.tables.setIamPolicy
bigquery.tables.update
bigquery.tables.updateData
bigquery.tables.updateTag
bigquery.transfers.get
bigquery.transfers.update
bigquery.jobs.create
bigquery.models.create
bigquery.models.getData
bigquery.models.updateData
bigquery.jobs.create
bigquery.models.create
bigquery.models.getData
bigquery.models.updateData
bigquery.models.updateMetadata
bigquery.models.delete
bigquery.jobs.create
bigquery.models.delete
bigquery.models.getMetadata
bigquery.models.list
bigquery.models.updateMetadata
bigquery.jobs.create
bigquery.models.getData
bigquery.jobs.create
bigquery.models.export
bigquery.models.updateTag

​

Ongoing permissions

Chalk’s support team will work with you to scope permissions down to what are needed for ongoing maintenance. The precise details depend on the level of ongoing support that your team needs and your compliance requirements. In principle, Chalk does not require ongoing access to data, or to the ability to edit IAM permissions, but Chalk requires ongoing access to update the software deployed in your environment.

​

Adding a Private Cloud SQL Data Source

If you would like to add a private Cloud SQL instance as a data source, this requires the deployment of a cloud-sql-proxy sidecar and connecting to the source like a regular PostgreSQL or MySQL source, with a hostname of 127.0.0.1.

First, ensure that Chalk’s customer-scoped management service account has the following IAM permissions for the instance and project in which your Cloud SQL source is set up.

Cloud SQL Client
Cloud SQL Instance User
Cloud SQL Viewer

Then, in the dashboard, navigate to Infrastructure > Resource Configuration and click on the Edit JSON button. Add the following to the resource configuration JSON:

​

Single Instance Configuration

For connecting to a single Cloud SQL instance:

{
  "sidecars": [
    {
      "name": "cloudsql",
      "instance_connection_name": "{PROJECT_NAME}:{REGION_NAME}:{INSTANCE_NAME}",
      "port": 5432,
      "http_port": 8097,
      "structured_logs": true,
      "http_address": "0.0.0.0",
      "private_ip": false,
      "psc": false,
      "auto_iam_authn": true
    }
  ]
}

This example creates a sidecar named cloudsql that connects to the specified instance on port 5432. You would then be able to connect to this sidecar by defining a chalk datasource with host http_address (0.0.0.0 in this example), and port port (5432 in this example).

​

Multiple Instances Configuration

For connecting to multiple Cloud SQL instances, use the instances map format:

{
  "sidecars": [
    {
      "name": "multi-cloudsql",
      "instances": {
        "{PROJECT_NAME}:{REGION_NAME}:{INSTANCE_1}": 5432,
        "{PROJECT_NAME}:{REGION_NAME}:{INSTANCE_2}": 3306
      },
      "http_port": 8097,
      "structured_logs": true,
      "http_address": "0.0.0.0",
      "private_ip": false,
      "psc": false,
      "auto_iam_authn": true
    }
  ]
}

This example creates a sidecar named cloudsql that connects to multiple specified instances on ports 5432 and 3306. You would then be able to connect to this sidecar by defining a chalk datasource with host http_address (0.0.0.0 in this example), and either port port (5432 or 3306 in this example).

​

Configuration Options

*Either instance_connection_name + port OR instances must be specified.

​

Advanced Configuration Options

​

Private IP Connection

Set "private_ip": true to connect to Cloud SQL instances using their private IP addresses instead of public IPs. This requires proper VPC connectivity.

​

Private Service Connect (PSC)

Set "psc": true to enable Private Service Connect for enhanced security and network isolation when connecting to Cloud SQL instances.

​

IAM Authentication

By setting "auto_iam_authn": true, the Cloud SQL proxy will automatically use IAM authentication to connect to your Cloud SQL instance. This eliminates the need to manage database passwords and leverages Google Cloud’s IAM service for secure authentication. When enabled, ensure that the service account used by Chalk has the appropriate Cloud SQL IAM database user permissions.

​

Example: Multiple Instances with All Options

{
  "sidecars": [
    {
      "name": "enterprise-cloudsql",
      "instances": {
        "prod-project:us-central1:postgres-primary": 5432,
        "prod-project:us-central1:mysql-analytics": 3306,
        "prod-project:europe-west1:postgres-replica": 5433
      },
      "http_port": 8097,
      "structured_logs": true,
      "http_address": "127.0.0.1",
      "private_ip": true,
      "psc": true,
      "auto_iam_authn": true
    }
  ]
}

Once you’re done editing the JSON, click on the Save and Apply Changes button to deploy your sidecar.

If you have any questions or need help, please reach out to Chalk’s support team.