Custom Attributes
We provide default attributes for every metric and span, but it’s often necessary to add extra context. To achieve this, we support both static and dynamic values based on headers.
Global Custom Attributes
Available since Router version 0.92.0.
Attributes based on request header values may potentially leak sensitive information when stored on the telemetry storage provider. You should only attach headers that have been sanitized by your infrastructure.
Each attribute increases the cardinality of a metric. Higher cardinality can be costly, as it increases the system’s memory usage and processing overhead.
To mitigate this, a limit of 2,000 unique combinations (variations) per metric is enforced. When this limit is exceeded, a new metric—without any attributes—is created and used instead. This approach helps maintain system stability and prevents excessive memory consumption.
There are scenarios where you may want to add additional attributes to your metrics and spans. One common use case is to add environment information, such as env=prod
. We provide an easy way to achieve this by adding a few lines of configuration.
Prometheus
All attributes are also added to the Prometheus metrics. Resource attributes are special and are only added as labels to the target_info
metric.
Static Attribute Value
This configuration will add the resource attribute env=prod
to every metric and span export. Additionally, the attribute service=static
is also added. Resource attributes identify the entity producing the traces and metrics. Since Prometheus metrics rely on OpenTelemetry metrics, the resource attributes are also added to the Prometheus target_info
series.
Dynamic Attribute Value
We also support setting attributes based on client request headers. To do this, simply specify the mapping in the value_from
property. If the headers could not be found, we will fallback to the default value. This functionality is only available to attributes
not resource_attributes
.
This will add the attribute service=static
unless the client has provided a request header x-service
, in which case the value of this header will be used instead of the default value. Remember to add the header to the allowlist in the CORS configuration; otherwise, CORS issues will occur on the frontend.
Expression Attributes
We also support using expressions for telemetry attributes (as well as metrics and tracing).
The above will attach the key “url_method” with the appropriate value to all spans and metrics. Note that when attaching values from expressions, if the value is not present at the time of evaluation it will not be attached.
Users can also use subgraph expressions in telemetry metrics as follows:
Unlike previously in this case, any expression with subgraph
in it, will only be evaluated for the “Engine - Fetch” span. Thus “sg_name” can only be found in the “Engine - Fetch” span.
You can find more information about expressions and the fields accessible for expressions here.
At the moment, we only allow expressions to return string values. If, for example, an expression returns an integer, you will need to cast it using exprlang’s string function.
Metrics Only Attributes
Available since Router version 0.126.0.
While using the telemetry
configuration allows users to specify attributes which will be attached to both spans AND metrics, there may be cases users want to attach values to metrics only. In this case you can use metrics only attributes.
The above configuration will ensure that the attribute service
will only be attached to metrics. Headers are pre-evaluated before metrics are emitted, while request context fields are determined at different stages. Consequently, request context fields may not be included in all metrics.
Additionally, the unique list of subgraph error codes will be appended to all emitted metric values. This enables correlation between error cases and corresponding metrics.
Tracing Only Attributes
Available since Router version VERSION.
While using the telemetry
configuration allows users to specify attributes which will be attached to both spans AND metrics, there may be cases users want to attach values to spans only.
In this case you can use trace only attributes.
The above configuration will ensure that the attribute service
will only be attached to spans.
For a complete list of available options, please refer to the relevant documentation section.