Observability
Search Chalk logs and similar pages (access logs, kube events, traces, and spans) with text terms, fields, Boolean operators, wildcards, and grouped values.
A query is composed of terms and operators. Terms can search the message or a specific field.
There are two types of terms:
timeout or chalkruntime."hello world".Use Boolean operators to combine terms. Operators are case sensitive and must be uppercase.
| Operator | Description | Example |
|---|---|---|
AND | Intersection: both terms must match. This is also the default between adjacent terms. | timeout AND service:chalkruntime |
OR | Union: either term can match. | service:chalkruntime OR service:api |
NOT | Exclusion: the following term or group must not match. | service:api NOT severity:ERROR |
- | Short exclusion syntax. | service:api -severity:ERROR |
A term without a field is matched against the record’s message as a substring, so timeout also
finds timeouts and connection_timeout_ms. Quotes are required to search a multi-word phrase.
| Search syntax | Description |
|---|---|
timeout | Searches message text for timeout. |
"request failed" | Searches message text for the phrase request failed. |
timeout error | Searches message text for both timeout and error. |
timeout OR error | Searches message text for either timeout or error. |
timeout and error | Searches for all three words. Lowercase and is a term, not an operator. |
"AND" | Searches for the literal word AND. Unquoted, it parses as the operator. |
To search a specific log field, use field:value.
service:chalkruntime
severity:ERROR
resolver:my_resolverUnlike a message term, a field value must match exactly unless it contains a wildcard. The available fields depend on which explorer you are in; the Logs fields are:
| Field | Description |
|---|---|
service | Service name. Also accepts service_name and ServiceName. |
component | Kubernetes or Chalk component. |
severity | TRACE, DEBUG, INFO, WARNING, ERROR, CRITICAL, FATAL. Case-insensitive, and warn and information are folded into their canonical spellings. |
resolver | Resolver name or FQN. Also accepts resolver_fqn. |
query_name | Named query. |
operation_id | Operation ID. |
correlation_id | Correlation ID. |
pod_name | Kubernetes pod name. |
app | Kubernetes app label. |
resource_group | Resource group. |
deployment_id | Deployment ID. Also accepts deployment. |
scaling_group | Scaling group label. |
container_name | Filter only. Container label. |
http_path | Request path, for services behind the proxy. |
response_code_class | 2xx, 4xx, 5xx — the response code bucketed by its first digit. |
user_logger | true for logs your resolvers emitted, false for Chalk’s own. |
trace_id | Trace ID. |
workflow_execution_id | Workflow execution ID. |
Fields that contain punctuation can be quoted:
"k8s.pod.name":engine-grpc* in a field value matches any run of characters:
| Search syntax | Matches |
|---|---|
pod_name:engine-* | Values beginning with engine-. |
pod_name:*abcdef | Values ending with abcdef. |
http_path:*VolumeService* | Values containing VolumeService. |
Put parentheses after a field name to apply that field to every unqualified value in the group.
service:(chalkruntime OR engine-grpc-mcyfr4j8u134)The query above is equivalent to:
service:chalkruntime OR service:engine-grpc-mcyfr4j8u134Groups can be nested and can include NOT:
service:((chalkruntime OR "engine grpc") AND NOT (proxy OR api))If a grouped value has its own field, Chalk keeps that explicit field:
service:(chalkruntime OR component:api)Use parentheses to control evaluation order:
(service:chalkruntime OR service:api) AND severity:ERRORWithout parentheses, the precedence order is:
NOT and -AND and adjacent termsORFor example:
"timeout" OR service:api AND NOT severity:ERRORis evaluated as:
"timeout" OR (service:api AND NOT severity:ERROR)Use double quotes when a value contains spaces, colons, parentheses, or other punctuation:
message:"key:value"
resource_name:"hello world"
service:"engine grpc"Inside a quoted value, escape a double quote with a backslash:
message:"failed with \"permission denied\""| Search query | Description |
|---|---|
service:chalkruntime severity:ERROR | Error logs from chalkruntime. |
service:(chalkruntime OR api) -severity:INFO | Logs from either service, excluding routine ones. |
resolver:my_resolver "timed out" | Resolver logs whose message contains timed out. |
(component:engine OR component:api) AND severity:ERROR | Error logs from either component. |
service:(NOT (proxy OR api)) timeout | Logs containing timeout from services other than proxy or api. |
pod_name:engine-* severity:ERROR | Error logs from any pod whose name starts with engine-. |
Every explorer (Logs, Access Logs, Kube Events, Spans, and Traces) accepts the syntax above. What differs between them is the useful attributes to query by.
Most fields can also be a group by dimension in
Search Aggregations. Where a field cannot, its description says
Filter only. The numeric measures — duration and count — are the exception: they are what the
aggregation row measures rather than groups by, so they appear there as fields for Mean, Percentile
and the rest.
| Field | Description |
|---|---|
authority | Target host, from the :authority header. |
http_path | Request path. Also accepts path. |
method | HTTP method, such as GET or POST. |
status_code | HTTP response code. Also accepts response_code. |
grpc_status | Status name, not the numeric code: OK, NOT_FOUND, UNAVAILABLE, and so on. |
response_flags | How Envoy terminated the request: - when nothing went wrong, else a code such as UH, UF or UT. |
upstream_host | IP address:port the request was proxied to, so this identifies the receiving pod. |
downstream_remote_address | IP address:port the request arrived from. The port is ephemeral, so match on the address with *. |
scaling_group | Scaling group label. Also accepts scaling_group_name. |
container | Container label. Also accepts container_name. |
pod_name | Kubernetes pod name. |
duration | Request duration, in milliseconds. The numeric measure the aggregation row offers for this surface. |
trace_id | Filter only. Trace ID, to pivot from a request to its spans. |
min_duration | Filter only. Lower bound on duration. A bare number is milliseconds; us, ms, s and m also work. |
max_duration | Filter only. Upper bound on duration, same units as min_duration. |
Both views filter the same spans over the fields below. Only the Spans view offers the aggregation row, so on Traces every field here is filter-only.
| Field | Description |
|---|---|
operation | Span name, such as a resolver or RPC name. |
service | Service that emitted the span, which for a client span is the caller rather than the callee. |
status_code | Span status: Ok, Error, or Unset. |
kind | Span kind, such as Server, Client, or Internal. |
trace | Trace ID. Groupable, so grouping by it counts how many spans each trace has. |
duration | Span duration, in nanoseconds. The numeric measure the aggregation row offers for this surface. |
span | Filter only. Span ID, to pull up one span and its children. |
min_duration | Filter only. Lower bound on duration. A bare number is microseconds; us, ms, s and m also work. |
max_duration | Filter only. Upper bound on duration, same units as min_duration. |
attr | Filter only. Span attribute, written attr:key=value. |
resource_attr | Filter only. Resource attribute, written resource_attr:key=value. |
| Field | Description |
|---|---|
severity | INFO, WARNING or ERROR — Chalk’s level, derived from the event rather than reported by it. |
reason | Kubernetes event reason, such as Unhealthy or FailedScheduling. |
name | Name of the object the event concerns, such as a pod or node name. |
kind | Kind of that object, such as Pod or Node. |
namespace | Namespace the object lives in. |
cluster_name | Cluster the event came from, for an environment spanning more than one. |
source_component | Component that reported the event, such as kubelet. |
event_type | Kubernetes’ own type, such as Warning, DiskPressure or Disruption. |
count | Occurrences Kubernetes coalesced into one event. The numeric measure the aggregation row offers here. |
pod_name | Filter only. Pod name — name narrowed to objects of kind Pod. |
message | Filter only. Substring match against the event message. |
Filtering queries can be combined with aggregations, to summarize data rather than list matching records. For more information, see Search Aggregations.