Aug 13: Context layer webinar
Chalk home page
  1. Queries
  2. Scheduled Query

When features are coming from a single SQL file, or a single resolver, you can use resolver crons to keep your online and offline stores up to date.

However, when features are chained together, or when you need to run a feature pipeline on a schedule, you can use scheduled queries.

Scheduled queries let you run an offline query on a schedule, and persist the results in the online and/or offline feature stores.

​
Create a scheduled query

To create a scheduled query, make a ScheduledQuery object somewhere in your code. Crontabs can be used to specify the schedule in the UTC timezone.

from chalk import ScheduledQuery

ScheduledQuery(
    name="enrich-transactions",
    schedule="0 0 * * *",
    output=[Transaction.clean_name, Transaction.category],
    store_online=True,
    store_offline=True,
)

When you run chalk apply, Chalk creates the scheduled query.

In the dashboard, open Offline > Schedules. The Definitions tab lists your scheduled queries with each one’s schedule, latest run, and next run, and the Scheduled query runs tab lists the runs themselves. Click a run’s ID to open the run, and use its Logs tab to see its logs. Click a scheduled query’s name to open its page, which has Overview, Run history, Scheduled run configuration, and Debug tabs.

​
Incrementalization

By default, scheduled queries use incrementalization to only ingest data which has been updated since the last run. You can also set a resolver to use as the source of the incrementalization. For example, if you were enriching financial transaction data, you might use the transactions.chalk.sql resolver as the source of incrementalization.

from chalk import ScheduledQuery

ScheduledQuery(
    name="enrich-transactions",
    schedule="0 0 * * *",
    output=[Transaction.clean_name, Transaction.category],
    store_online=True,
    store_offline=True,
    incremental_resolvers="transactions",
)

​
Monitoring

To get notified when a scheduled query fails, create a metric chart monitor with these settings:

  • Series: Cron Runs, filtered to Resolver Status failure
  • Threshold: Above 0
  • Alert channels: the Slack, PagerDuty, or incident.io channels to notify

See Monitors for how monitors, incidents, and alert channels work.

Additionally, you can export metrics about scheduled query executions to your observability tools. Use scheduled_query_count (scheduled_query_count_total on /v2/metrics/export) to count executions by query_name and query_status. Filter on query_status="failure" and group by query_name to alert on failures for each scheduled query. You can also export scheduled_query_success_ratio for the fraction of executions that succeeded per query. Both metrics cover executions since midnight UTC. See the metrics export documentation for labels and counter reset behavior.

​
Resources

By default, scheduled query runs are picked up by the Job Queue Consumer. To see your Job Queue Consumer resource configurations, check the Job Queue Consumer service under Resources in the dashboard. To customize the resources allocated to runs for a specific scheduled query, set the resource configuration under the Scheduled run configuration tab on the scheduled query’s page in the dashboard.