# Search Syntax source: https://docs.chalk.ai/docs/log-search-syntax ## Search Chalk logs and similar pages (access logs, kube events, traces, and spans) with text terms, fields, Boolean operators, wildcards, and grouped values. A query is composed of terms and operators. Terms can search the message or a specific field. There are two types of terms: - A single term is one unquoted word, such as timeout or chalkruntime. - A phrase is text surrounded by double quotes, such as "hello world". Use Boolean operators to combine terms. Operators are case sensitive and must be uppercase. | Operator | Description | Example | | -------- | ------------------------------------------------------------------------------------- | ------------------------------------- | | `AND` | Intersection: both terms must match. This is also the default between adjacent terms. | `timeout AND service:chalkruntime` | | `OR` | Union: either term can match. | `service:chalkruntime OR service:api` | | `NOT` | Exclusion: the following term or group must not match. | `service:api NOT severity:ERROR` | | `-` | Short exclusion syntax. | `service:api -severity:ERROR` | ### Text Search A term without a field is matched against the record's message as a substring, so timeout also finds timeouts and connection_timeout_ms. Quotes are required to search a multi-word phrase. | Search syntax | Description | | ------------------- | ------------------------------------------------------------------------- | | `timeout` | Searches message text for `timeout`. | | `"request failed"` | Searches message text for the phrase `request failed`. | | `timeout error` | Searches message text for both `timeout` and `error`. | | `timeout OR error` | Searches message text for either `timeout` or `error`. | | `timeout and error` | Searches for all three words. Lowercase `and` is a term, not an operator. | | `"AND"` | Searches for the literal word `AND`. Unquoted, it parses as the operator. | ### Field Search To search a specific log field, use field:value. ``` service:chalkruntime severity:ERROR resolver:my_resolver ``` Unlike a message term, a field value must match exactly unless it contains a wildcard. The available fields depend on which explorer you are in; the Logs fields are: | Field | Description | | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `service` | Service name. Also accepts `service_name` and `ServiceName`. | | `component` | Kubernetes or Chalk component. | | `severity` | `TRACE`, `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, `FATAL`. Case-insensitive, and `warn` and `information` are folded into their canonical spellings. | | `resolver` | Resolver name or FQN. Also accepts `resolver_fqn`. | | `query_name` | Named query. | | `operation_id` | Operation ID. | | `correlation_id` | Correlation ID. | | `pod_name` | Kubernetes pod name. | | `app` | Kubernetes app label. | | `resource_group` | Resource group. | | `deployment_id` | Deployment ID. Also accepts `deployment`. | | `scaling_group` | Scaling group label. | | `container_name` | **Filter only.** Container label. | | `http_path` | Request path, for services behind the proxy. | | `response_code_class` | `2xx`, `4xx`, `5xx` — the response code bucketed by its first digit. | | `user_logger` | `true` for logs your resolvers emitted, `false` for Chalk's own. | | `trace_id` | Trace ID. | | `workflow_execution_id` | Workflow execution ID. | Fields that contain punctuation can be quoted: ``` "k8s.pod.name":engine-grpc ``` ### Wildcards * in a field value matches any run of characters: | Search syntax | Matches | | --------------------------- | ---------------------------------- | | `pod_name:engine-*` | Values beginning with `engine-`. | | `pod_name:*abcdef` | Values ending with `abcdef`. | | `http_path:*VolumeService*` | Values containing `VolumeService`. | ### Grouped Field Values Put parentheses after a field name to apply that field to every unqualified value in the group. ``` service:(chalkruntime OR engine-grpc-mcyfr4j8u134) ``` The query above is equivalent to: ``` service:chalkruntime OR service:engine-grpc-mcyfr4j8u134 ``` Groups can be nested and can include NOT: ``` service:((chalkruntime OR "engine grpc") AND NOT (proxy OR api)) ``` If a grouped value has its own field, Chalk keeps that explicit field: ``` service:(chalkruntime OR component:api) ``` ### Grouping And Precedence Use parentheses to control evaluation order: ``` (service:chalkruntime OR service:api) AND severity:ERROR ``` Without parentheses, the precedence order is: - NOT and - - AND and adjacent terms - OR For example: ``` "timeout" OR service:api AND NOT severity:ERROR ``` is evaluated as: ``` "timeout" OR (service:api AND NOT severity:ERROR) ``` ### Special Characters And Spaces Use double quotes when a value contains spaces, colons, parentheses, or other punctuation: ``` message:"key:value" resource_name:"hello world" service:"engine grpc" ``` Inside a quoted value, escape a double quote with a backslash: ``` message:"failed with \"permission denied\"" ``` ### Examples | Search query | Description | | -------------------------------------------------------- | -------------------------------------------------------------------- | | `service:chalkruntime severity:ERROR` | Error logs from `chalkruntime`. | | `service:(chalkruntime OR api) -severity:INFO` | Logs from either service, excluding routine ones. | | `resolver:my_resolver "timed out"` | Resolver logs whose message contains `timed out`. | | `(component:engine OR component:api) AND severity:ERROR` | Error logs from either component. | | `service:(NOT (proxy OR api)) timeout` | Logs containing `timeout` from services other than `proxy` or `api`. | | `pod_name:engine-* severity:ERROR` | Error logs from any pod whose name starts with `engine-`. | ### Fields By Surface Every explorer (Logs, Access Logs, Kube Events, Spans, and Traces) accepts the syntax above. What differs between them is the useful attributes to query by. Most fields can also be a group by dimension in Search Aggregations. Where a field cannot, its description says Filter only. The numeric measures — duration and count — are the exception: they are what the aggregation row measures rather than groups by, so they appear there as fields for Mean, Percentile and the rest. ### Access Logs | Field | Description | | --------------------------- | ----------------------------------------------------------------------------------------------------------- | | `authority` | Target host, from the `:authority` header. | | `http_path` | Request path. Also accepts `path`. | | `method` | HTTP method, such as `GET` or `POST`. | | `status_code` | HTTP response code. Also accepts `response_code`. | | `grpc_status` | Status name, not the numeric code: `OK`, `NOT_FOUND`, `UNAVAILABLE`, and so on. | | `response_flags` | How Envoy terminated the request: `-` when nothing went wrong, else a code such as `UH`, `UF` or `UT`. | | `upstream_host` | IP `address:port` the request was proxied to, so this identifies the receiving pod. | | `downstream_remote_address` | IP `address:port` the request arrived from. The port is ephemeral, so match on the address with `*`. | | `scaling_group` | Scaling group label. Also accepts `scaling_group_name`. | | `container` | Container label. Also accepts `container_name`. | | `pod_name` | Kubernetes pod name. | | `duration` | Request duration, in milliseconds. The numeric measure the aggregation row offers for this surface. | | `trace_id` | **Filter only.** Trace ID, to pivot from a request to its spans. | | `min_duration` | **Filter only.** Lower bound on duration. A bare number is milliseconds; `us`, `ms`, `s` and `m` also work. | | `max_duration` | **Filter only.** Upper bound on duration, same units as `min_duration`. | ### Spans and Traces Both views filter the same spans over the fields below. Only the Spans view offers the aggregation row, so on Traces every field here is filter-only. | Field | Description | | --------------- | ----------------------------------------------------------------------------------------------------------- | | `operation` | Span name, such as a resolver or RPC name. | | `service` | Service that emitted the span, which for a client span is the caller rather than the callee. | | `status_code` | Span status: `Ok`, `Error`, or `Unset`. | | `kind` | Span kind, such as `Server`, `Client`, or `Internal`. | | `trace` | Trace ID. Groupable, so grouping by it counts how many spans each trace has. | | `duration` | Span duration, in nanoseconds. The numeric measure the aggregation row offers for this surface. | | `span` | **Filter only.** Span ID, to pull up one span and its children. | | `min_duration` | **Filter only.** Lower bound on duration. A bare number is microseconds; `us`, `ms`, `s` and `m` also work. | | `max_duration` | **Filter only.** Upper bound on duration, same units as `min_duration`. | | `attr` | **Filter only.** Span attribute, written `attr:key=value`. | | `resource_attr` | **Filter only.** Resource attribute, written `resource_attr:key=value`. | ### Kube Events | Field | Description | | ------------------ | ----------------------------------------------------------------------------------------------------- | | `severity` | `INFO`, `WARNING` or `ERROR` — Chalk's level, derived from the event rather than reported by it. | | `reason` | Kubernetes event reason, such as `Unhealthy` or `FailedScheduling`. | | `name` | Name of the object the event concerns, such as a pod or node name. | | `kind` | Kind of that object, such as `Pod` or `Node`. | | `namespace` | Namespace the object lives in. | | `cluster_name` | Cluster the event came from, for an environment spanning more than one. | | `source_component` | Component that reported the event, such as `kubelet`. | | `event_type` | Kubernetes' own type, such as `Warning`, `DiskPressure` or `Disruption`. | | `count` | Occurrences Kubernetes coalesced into one event. The numeric measure the aggregation row offers here. | | `pod_name` | **Filter only.** Pod name — `name` narrowed to objects of kind `Pod`. | | `message` | **Filter only.** Substring match against the event message. | ### Aggregations Filtering queries can be combined with aggregations, to summarize data rather than list matching records. For more information, see Search Aggregations.