Observability
Search Chalk logs with text terms, facets, Boolean operators, and grouped values.
Chalk log search is composed of terms and operators. Terms can search the log message or a specific log field.
There are two types of terms:
timeout or chalkruntime."hello world".Use Boolean operators to combine terms. Operators are case sensitive and must be uppercase.
| Operator | Description | Example |
|---|---|---|
AND | Intersection: both terms must match. This is also the default between adjacent terms. | timeout AND service:chalkruntime |
OR | Union: either term can match. | service:chalkruntime OR service:api |
NOT | Exclusion: the following term or group must not match. | service:api NOT status:error |
- | Short exclusion syntax. | service:api -status:error |
A term without a field searches the log message.
| Search syntax | Description |
|---|---|
timeout | Searches message text for timeout. |
"request failed" | Searches message text for the phrase request failed. |
timeout error | Searches message text for both timeout and error. |
timeout OR error | Searches message text for either timeout or error. |
If you need to search for a word that is also an operator, quote it:
"AND"To search a specific log field, use field:value.
service:chalkruntime
status:error
resolver:my_resolverField search uses exact field names and values. The available fields depend on the log source, but common fields include:
| Field | Description |
|---|---|
service | Service name. Also accepts service_name and ServiceName. |
component | Kubernetes or Chalk component. |
status | Status value when present in the log record. |
severity | Log level, such as debug, info, warn, or error. |
resolver | Resolver name or FQN. Also accepts resolver_fqn. |
query_name | Named query. |
operation_id | Operation ID. |
correlation_id | Correlation ID. |
pod_name | Kubernetes pod name. |
app | Kubernetes app label. |
resource_group | Resource group. |
deployment | Deployment ID. Also accepts deployment_id. |
scaling_group | Scaling group label. |
container | Container label. |
trace_id | Trace ID. |
workflow_execution_id | Workflow execution ID. |
Fields that contain punctuation can be quoted:
"k8s.pod.name":engine-grpcPut parentheses after a field name to apply that field to every unqualified value in the group.
service:(chalkruntime OR engine-grpc-mcyfr4j8u134)The query above is equivalent to:
service:chalkruntime OR service:engine-grpc-mcyfr4j8u134Groups can be nested and can include NOT:
service:((chalkruntime OR "engine grpc") AND NOT (proxy OR api))If a grouped value has its own field, Chalk keeps that explicit field:
service:(chalkruntime OR component:api)Use parentheses to control evaluation order:
(service:chalkruntime OR service:api) AND status:errorWithout parentheses, the precedence order is:
NOT and -AND and adjacent termsORFor example:
"timeout" OR service:api AND NOT status:erroris evaluated as:
"timeout" OR (service:api AND NOT status:error)Use double quotes when a value contains spaces, colons, parentheses, or other punctuation:
message:"key:value"
resource_name:"hello world"
service:"engine grpc"Inside a quoted value, escape a double quote with a backslash:
message:"failed with \"permission denied\""| Search query | Description |
|---|---|
service:chalkruntime status:error | Logs from chalkruntime with status error. |
service:(chalkruntime OR api) -status:ok | Logs from either service, excluding successful records. |
resolver:my_resolver "timed out" | Resolver logs whose message contains timed out. |
(component:engine OR component:api) AND severity:error | Error logs from either component. |
service:(NOT (proxy OR api)) timeout | Logs containing timeout from services other than proxy or api. |
Chalk log search does not currently support Datadog-only syntax such as @attribute prefixes,
numeric ranges like [400 TO 499], CIDR(), calculated fields, or existence queries such as
field:*.