Offline Query Heartbeat Timeout

Offline queries can consume significant amounts of system resources. While an offline query runs, it sends regular heartbeats to Chalk. If a query stops sending heartbeats for several minutes, Chalk marks it as failed and stops it. The query’s error then says that it “failed from missing a heartbeat. This means it probably encountered an out-of-memory error.”

Typically, this is due to the query being too resource-intensive for the system to handle. If you are experiencing offline query timeouts, you may want to consider the following:

0. Check for out-of-memory errors: Check the ‘metrics’ tab of your query, and look at the reported memory utilization. If you are running out of memory, you may want to increase the memory requests for your offline query pods. You can do this by navigating to the “Resources” page in the Chalk UI and increasing the memory requests for offline query workers.
1. Use run_asynchronously: Add run_asynchronously=True to isolate your query to its own Kubernetes pod. This lowers the chance that your query will contend with other workloads on a single machine and may help prevent timeouts.
2. Adjust node selectors: If you are using node selectors to run your offline queries on specific nodes, ensure that the nodes you are selecting have enough resources to handle the query. Make sure that your workloads are not co-scheduled with other intense tasks.
3. Shard your query: If you are running a large query, consider breaking it into smaller pieces using num_shards. This can help prevent timeouts and improve the overall performance of your query.
4. Optimize your query: Look at the operations the query is processing. Check for Python resolvers that use heavy libraries, or that perform memory-intensive operations like Pandas transformations.

If you are still experiencing timeouts, contact Chalk Support for further assistance. Please provide an example failed query and context on how to reproduce the issue.