Relabeling cookbook

VictoriaMetrics and vmagent support Prometheus-compatible relabeling with additional enhancements.

The relabeling is mostly used for the following tasks:

Dropping unneeded scrape targets during service discovery. See how to drop discovered targets.
Adding or updating static labels at scrape targets. See how to add labels to scrape targets.
Copying target labels from another labels. See how to copy labels in scrape targets.
Modifying scrape urls for discovered targets. See how to modify scrape urls in targets.
Modifying instance and job labels. See how to modify instance and job.
Extracting label parts into another labels. See how to extract label parts.
Removing prefixes from target label names. See how to remove prefixes from target label names.
Removing some labels from discovered targets. See how to remove labels from targets.
Dropping some metrics during scape. See how to drop metrics during scrape.
Removing some labels from scraped metrics. See how to remove labels from metrics.
Removing some labels from metrics matching some series selector. See how to remove labels from metrics subset.

How to remove labels from metrics subset

Sometimes it may be needed to remove labels from a subset of scraped metrics, while leaving these labels in the rest of scraped metrics. In this case the if series selector can be applied to action: labeldrop or action: labelkeep.

For example, the following config drops labels with names starting from foo_ prefix from metrics matching a{b="c"} series selector:

scrape_configs:
- job_name: test
  static_configs:
  - targets: [host123]
  metric_relabel_configs:
  - action: labeldrop
    if: 'a{b="c"}'
    regex: "foo_.*"
  

How to remove labels from scraped metrics

Sometimes it may be needed to remove labels from scraped metrics. For example, if some labels lead to high cardinality or high churn rate issues, then it may be a good idea to drop these labels during scrapes. This can be done with action: labeldrop or action: labelkeep relabeling rules at metric_relabel_configs section:

action: labeldrop drops labels with names matching the given regex option
action: labelkeep drops labels with names not matching the given regex option

For example, the following config drops labels with names starting with foo_ prefix from all the metrics scraped from the http://host123/metrics:

scrape_configs:
- job_name: test
  static_configs:
  - targets: [host123]
  metric_relabel_configs:
  - action: labeldrop
    regex: "foo_.*"
  

Important notes:

Labels with __ prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules.
Make sure that metrics exposed by the target can be uniquely identified by their names and the remaining labels after label removal. Otherwise, duplicate metrics with duplicate timestamps and different values will be pushed to the storage. This is an undesired issue in most cases.

How to drop metrics during scrape

Sometimes it is needed to drop some metrics during scrapes. For example, if some metrics result in high cardinality or high churn rate issues, then it may be a good idea to drop these metrics during scrapes. This can be done with the action: drop or action: keep relabeling rules at metric_relabel_configs section:

action: drop drops all the metrics, which match the if series selector
action: keep drops all the metrics, which don't match the if series selector

For example, the following config drops all the metrics obtained from http://host123/metrics, which start with foo_ prefix:

scrape_configs:
- job_name: test
  static_configs:
  - targets: [host123]
  metric_relabel_configs:
  - if: '{__name__=~"foo_.*"}'
    action: drop
  

Note that the relabeling config is specified under metric_relabel_configs section instead of relabel_configs section:

The relabel_configs is applied to the configured/discovered targets.
The metric_relabel_configs is applied to metrics scraped from the configured/discovered targets.

How to remove labels from a subset of targets

Sometimes it is needed to remove some labels from a subset of discovered targets, while leaving these labels in the rest of discovered targets. In this case the if selector can be added to action: labeldrop or action: labelkeep relabeling rule.

For example, the following config discovers pod targets in Kubernetes, extracts pod-level labels into labels with foo_ prefix and then drops all the labels with foo_bar_ prefix in their names for targets matching {__address__=~"pod123.+"} selector:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: labelmap
    regex: "__meta_kubernetes_pod_label_(.+)"
    replacement: "foo_$1"
  - action: labeldrop
    if: '{__address__=~"pod123.+"}'
    regex: "foo_bar_.*"
  

How to remove labels from targets

Sometimes it is needed to remove some labels from discovered targets. In this case the action: labeldrop and action: labelkeep relabeling options can be used:

action: labeldrop drops all the labels with names matching the regex option
action: labelkeep drops all the labels with names not matching the regex option

For example, the following config discovers pod targets in Kubernetes, extracts pod-level labels into labels with foo_ prefix and then drops all the labels with foo_bar_ prefix in their names:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: labelmap
    regex: "__meta_kubernetes_pod_label_(.+)"
    replacement: "foo_$1"
  - action: labeldrop
    regex: "foo_bar_.*"
  

Important notes:

Labels with __ prefix are automatically removed after the relabeling, so there is no need in removing them with relabeling rules.
Do not remove instance and job labels, since this may result in duplicate scrape targets with identical sets of labels.

How to remove prefixes from target label names

Sometimes it is needed to remove __meta_* prefixes from meta-labels of the discovered targets. For example, Kubernetes service discovery adds __meta_kubernetes_pod_label_<labelname> labels per each pod-level label. In this case it may be needed to leave only the <labelname> part of such label names, while removing the __meta_kubernetes_pod_label_ prefix. This can be done with action: labelmap relabeling option:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - action: labelmap
    regex: "__meta_kubernetes_pod_label_(.+)"
    replacement: "$1"
  

The regex option can contain arbitrary regular expression in RE2 format. The regex option is applied to every label name in the target. It is automatically anchored, so it must match the whole label name. It can contain capture groups such as (.+) in the config above. These capture groups can be referenced then inside replacement option with the $N syntax, where N is the number of the capture group in regex. The first capture group has the $1 reference.

The label name is left as is if the regex doesn't match it.

How to extract label parts

Relabeling allows extracting parts from label values and storing them into arbitrary labels. This is performed with regex and replacement options in relabeling rules.

For example, the following config discovers pod targets in Kubernetes, extracts bar part from foo/bar container name and stores it into the xyz label with abc_ prefix:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_container_name]
    regex: "[^/]+/(.+)"
    replacement: "abc_$1"
    target_label: xyz
  

The regex option can contain arbitrary regular expression in RE2 format. The regex option is automatically anchored, so it must match the whole value from source_labels. It can contain capture groups such as (.+) in the config above. These capture groups can be referenced then inside replacement option with the $N syntax, where N is the number of the capture group in regex. The first capture group has the $1 reference.

It is possible to construct a label from multiple parts of different labels. In this case just specify the needed source labels inside source_labels list. The values of labels specified in source_labels list are joined with ; separator by default before being matched against the regex. The separator can be overridden via separator option.

If the regex doesn't match the value constructed from source_labels, then the relabeling rule is skipped and the remaining relabeling rules are executed.

How to modify instance and job

Single-node VictoriaMetrics and vmagent automatically add instance and job labels per each discovered target:

The job label is set to job_name value specified in the corresponding scrape_config.
The instance label is set to the host:port part of __address__ label value after target-level relabeling. The __address__ label value is automatically set to the most suitable value depending on the used service discovery type. The __address__ label can be overridden during relabeling - see these docs.

Both instance and job labels can be overridden during relabeling. For example, the following config discovers pod targets in Kubernetes and overrides job label from k8s to foo:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - target_label: job
    replacement: foo
  

How to modify scrape urls in targets

URLs for scrape targets are composed of the following parts:

Scheme (e.g. http or https). The scheme is available during target relabeling in a special label - __scheme__. By default, the scheme is set to http. It can be overridden either by specifying the scheme option at scrape_config level or by updating the __scheme__ label during relabeling.
Host and port (e.g. host12:3456). This information is available during target relabeling in a special label - __address__. Every supported service discovery type sets the __address__ label to the most suitable value. Sometimes this value needs to be modified. In this case just update the __address__ label during relabeling to the needed value. The port part is optional. If it is missing, then it is automatically set either to 80 or 443 depending on the used scheme (http or https). The host:port part from the final __address__ label is automatically set to instance label unless the instance label is explicitly set during relabeling. The __address__ label can contain the full scrape url, e.g. http://host:port/metrics/path?query_args. In this case the __scheme__ and __metrics_path__ labels are ignored.
URL path (e.g. /metrics). This information is available during target relabeling in a special label - __metrics_path__. By default, the __metrics_path__ is set to /metrics. It can be overridden either by specifying the metrics_path option at scrape_config or by updating the __metrics_path__ label during relabeling.
Query args (e.g. ?foo=bar&baz=xyz). This information is available during target relabeling in special labels with __param_ prefix. For example, __param_foo would have the bar value, while __param_baz would have the xyz value for ?foo=bar&baz=xyz query string. The query args can be specified either via params section at scrape_config or by updating/setting the corresponding __param_* labels during relabeling.

The resulting scrape url looks like the following:

  <__scheme__> + "://" + <__address__> + <__metrics_path__> + <"?" + query_args_from_param_labels>
  

It is expected that the target exposes metrics in Prometheus text exposition format at the resulting scrape url.

Given the scrape url construction rules above, the following config discovers pod targets in Kubernetes and constructs per-target scrape url as https://<pod_name>/foo/bar?baz=<container_name>:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  metrics_path: /foo/bar
  relabel_configs:
  - target_label: __scheme__
    replacement: https
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: __address__
  - source_labels: [__meta_kubernetes_pod_container_name]
    target_label: __param_baz
  

How to copy labels in scrape targets

Labels can be copied by specifying the source labels via source_labels relabeling option and specifying the target label via target_label relabeling option. For example, the following config copies __meta_kubernetes_pod_name label to pod label for all the discovered pods in Kubernetes:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod
  

Note that the source_labels option accepts a list of labels in square brackets. If multiple labels are specified in the source_labels list, then the specified label values are joined into a single string with ; delimiter by default. The delimiter can be modified by specifying it via separator option. For example, the following config sets the pod_name:container_port value to the host_port label for all the discovered pod targets in Kubernetes:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_name, __meta_kubernetes_pod_container_port_number]
    separator: ":"
    target_label: host_port
  

How to add labels to scrape targets

Additional labels can be added to scrape targets by specifying the label name in target_label relabeling option and by specifying the label value in replacement relabeling option. The same approach can be used for updating already existing label values at target level.

For example, the following config adds {foo="bar"} label to all the discovered pods in Kubernetes:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - target_label: "foo"
    replacement: "bar"
  

The labels, which are added to the target, are automatically added to all the metrics scraped from the target. For example, if the target exposes the metric metric{label="value"}, then the metric is transformed into metric{label="value",foo="bar"} before being sent to the storage.

If the metric exported by the target contains the same label as the target itself, then the exported_ prefix is added to the exported label name. For example, if the target exposes the metric metric{foo="baz"}, then the metric is transformed into metric{exported_foo="baz",foo="bar"}. This behaviour can be changed by specifying honor_labels: true option at the given scrape config. In this case the exported label overrides the target's label. In this case the metric{foo="baz"} stays the same. Example config with honor_labels: true:

scrape_configs:
- job_name: k8s
  kubernetes_sd_configs:
  - role: pod
  honor_labels: true
  relabel_configs:
  - target_label: "foo"
    replacement: "bar"
  

How to drop discovered targets

If a particular discovered target shouldn't be scraped, then action: keep or action: drop relabeling rules must be used inside relabel_configs section.

The action: keep keeps only scrape targets with labels matching the if selector, while dropping the rest of targets. For example, the following config discovers pod targets in Kubernetes and scrapes only pods with names starting with foo prefix:

scrape_configs:
- job_name: foo_pods
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - if: '{__meta_kubernetes_pod_name=~"foo.*"}'
    action: keep
  

The action: drop drops all the scrape targets with labels matching the if selector, while keeping the rest of targets. For example, the following config discovers pod targets in Kubernetes and scrapes only pods with names starting with prefixes other than foo:

scrape_configs:
- job_name: not_foo_pods
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - if: '{__meta_kubernetes_pod_name=~"foo.*"}'
    action: drop
  

Useful tips for target relabeling

Target relabeling can be debugged by clicking the debug link for the needed target on the http://vmagent:8429/target or on the http://vmagent:8429/service-discovery pages. See these docs.
Every discovered target contains a set of meta-labels, which start with __meta_ prefix. The specific sets of labels per each supported service discovery option are listed here.
Every discovered target contains additional labels with __ prefix other than __meta_ labels. See these docs for more details.
All the labels, which start with __ prefix, are automatically removed from targets after the relabeling. So it is common practice to store temporary labels with names starting with __ during target relabeling.
All the target-level labels are automatically added to all the metrics scraped from targets.
The list of discovered scrape targets with all the discovered meta-labels is available at http://vmagent:8429/service-discovery page for vmagent and at http://victoriametrics:8428/service-discovery page for single-node VictoriaMetrics.
The list of active targets with the final set of labels left after relabeling is available at http://vmagent:8429/targets page for vmagent and at http://victoriametrics:8428/targets page for single-node VictoriaMetrics.

Useful tips for metric relabeling

Metric relabeling can be debugged at http://vmagent:8429/metric-relabel-debug page. See these docs.
All the labels, which start with __ prefix, are automatically removed from metrics after the relabeling. So it is common practice to store temporary labels with names starting with __ during metrics relabeling.
All the target-level labels are automatically added to all the metrics scraped from targets, so target-level labels are available during metrics relabeling.