CHANGELOG

The following tip changes can be tested by building VictoriaMetrics components from the latest commits according to the following docs:

tip

FEATURE: vmbackup: add support for server-side copy of existing backups. See these docs for details.
FEATURE: vmagent: properly handle unexpected EOF error when parsing metrics in Prometheus exposition format. See this issue.
FEATURE: vmagent: retry failed write request on the closed connection immediately, without waiting for backoff. This should improve data delivery speed and reduce amount of error logs emitted by vmagent when using idle connections. See related issue.
BUGFIX: do not allow starting VictoriaMetrics components with improperly set boolean command-line flags in the form -boolFlagName value, since this leads to silent incomplete flags' parsing. This form should be replaced with -boolFlagName=value. See this issue.
BUGFIX: vmagent: properly set labels from -remoteWrite.label command-line flag just before sending samples to the configured -remoteWrite.url according to these docs. Previously these labels were incorrectly set before the relabeling configured via -remoteWrite.urlRelabelConfigs and the stream aggregation configured via -remoteWrite.streamAggr.config, so these labels could be lost or incorrectly transformed before sending the samples to remote storage. The fix allows using -remoteWrite.label for identifying vmagent instances in cluster mode. See this issue and these docs for more details.
BUGFIX: remove DEBUG logging when parsing if filters inside relabeling rules and when parsing match filters inside stream aggregation rules.
BUGFIX: properly replace : chars in label names with _ when -usePromCompatibleNaming command-line flag is passed to vmagent, vminsert or single-node VictoriaMetrics. This addresses this comment.
BUGFIX: Official Grafana dashboards for VictoriaMetrics: fix display of ingested rows rate for Samples ingested/s and Samples rate panels for vmagent's dasbhoard. Previously, not all ingested protocols were accounted in these panels. An extra panel Rows rate was added to Ingestion section to display the split for rows ingested rate by protocol.
BUGFIX: vmbackup: correctly check if specified -dst belongs to specified -storageDataPath. See this issue.
BUGFIX: vmctl: don't interrupt the migration process if no metrics were found for a specific tenant. See this issue.

v1.93.0

Released at 2023-08-12

v1.93.x is a line of LTS releases (e.g. long-time support). It contains important up-to-date bugfixes. The v1.93.x line will be supported for at least 12 months since v1.93.0 release

Update note: starting from this release, vmagent ignores timestamps provided by scrape targets by default - it associates scraped metrics with local timestamps instead. Set honor_timestamps: true in scrape configs if timestamps provided by scrape targets must be used instead. This change helps removing gaps for metrics collected from cadvisor such as container_memory_usage_bytes. This also improves data compression and query performance over metrics collected from cadvisor. See more details here.

SECURITY: upgrade Go builder from Go1.20.6 to Go1.21.0 in order to fix this issue.
SECURITY: upgrade base docker image (Alpine) from 3.18.2 to 3.18.3. See alpine 3.18.3 release notes.
FEATURE: MetricsQL: add share_eq_over_time(m[d], eq) function for calculating the share (in the range [0...1]) of raw samples on the given lookbehind window d, which are equal to eq. See this feature request. Thanks to @Damon07 for the pull request.
FEATURE: vmauth: allow configuring deadline for a backend to be excluded from the rotation on errors via -failTimeout cmd-line flag. This feature could be useful when it is expected for backends to be not available for significant periods of time. See this issue for details. Thanks to @SunKyu for the pull request.
FEATURE: vmalert: remove deprecated in v1.61.0 -rule.configCheckInterval command-line flag. Use -configCheckInterval command-line flag instead.
FEATURE: vmalert: remove support of deprecated web links of /api/v1/<groupID>/<alertID>/status form in favour of /api/v1/alerts?group_id=<>&alert_id=<> links. Links of /api/v1/<groupID>/<alertID>/status form were deprecated in v1.79.0. See this issue for details.
FEATURE: vmctl: allow disabling binary export API protocol via -vm-native-disable-binary-protocol cmd-line flag when migrating data from VictoriaMetrics. Disabling binary protocol can be useful for deduplication of the exported data before ingestion. For this, deduplication need to be configured at -vm-native-src-addr side and -vm-native-disable-binary-protocol should be set on vmctl side.
FEATURE: vmctl: add support of week step for time-based chunking migration. See this issue.
FEATURE: vmctl: allow specifying custom full url at --remote-read-src-addr command-line flag if --remote-read-disable-path-append command-line flag is set. This allows importing data from urls, which do not end with /api/v1/read. For example, from Promscale. See this issue.
FEATURE: vmui: add warning in query field of vmui for partial data responses. See this issue.
FEATURE: vmui: allow displaying the full error message on click for trimmed error messages in vmui. See this issue.
FEATURE: Official Grafana dashboards for VictoriaMetrics: add Concurrent inserts panel to vmagent's dasbhoard. The new panel supposed to show whether the number of concurrent inserts processed by vmagent isn't reaching the limit.
FEATURE: Official Grafana dashboards for VictoriaMetrics: add panels for absolute Mem and CPU usage by vmalert. See related issue here.
FEATURE: Official Grafana dashboards for VictoriaMetrics: correctly calculate Bytes per point value for single-server and cluster VM dashboards. Before, the calculation mistakenly accounted for the number of entries in indexdb in denominator, which could have shown lower values than expected.
FEATURE: Alerting rules for VictoriaMetrics: ConcurrentFlushesHitTheLimit alerting rule was moved from single-server and cluster alerts to the list of "health" alerts as it could be related to many VictoriaMetrics components.
BUGFIX: vmagent: return human readable error if opentelemetry has json encoding. Follow-up after PR.
BUGFIX: vmagent: properly validate scheme for proxy_url field at the scrape config. See this issue for details.
BUGFIX: vmagent: properly apply if filters during relabeling. Previously the if filter could improperly work. See this issue and this pull request.
BUGFIX: vmagent: use local scrape timestamps for the scraped metrics unless honor_timestamps: true option is explicitly set at scrape_config. This fixes gaps for metrics collected from cadvisor or similar exporters, which export metrics with invalid timestamps. See this issue and this comment for details. The issue has been introduced in v1.68.0.
BUGFIX: vmagent: fixes runtime panic at OpenTelemetry parser. Opentelemetry format allows histograms without sum fields. Such histogram converted as counter with _count suffix. See this issue.
BUGFIX: vmagent: keep unmatched series at stream aggregation when -remoteWrite.streamAggr.dropInput is set to false to match intended behaviour introduced at v1.92.0. See this issue.
BUGFIX: vmalert: properly set vmalert_config_last_reload_successful value on configuration updates or rollbacks. The bug was introduced in v1.92.0 in this PR.
BUGFIX: vmalert: fix vmalert_remotewrite_send_duration_seconds_total value, before it didn't count in the real time spending on remote write requests. See this pr for details.
BUGFIX: vmbackupmanager: fix panic when creating a backup to a local filesystem on Windows. See this issue.
BUGFIX: vmui: properly handle client address with X-Forwarded-For part at the Active queries page. See this comment.
BUGFIX: MetricsQL: prevent from panic when the lookbehind window in square brackets of rollup function is parsed into negative value. See this issue.

v1.92.1

Released at 2023-07-28

BUGFIX: vmalert: revert unit test feature for alerting and recording rules introduced in this pull request. See the following change.

v1.92.0

Released at 2023-07-27

Update note: starting from this release, stream aggregation writes the following samples to the configured remote storage by default:

aggregated samples;
the original input samples, which match zero match options from the provided config.

Previously only aggregated samples were written to the storage by default. The previous behavior can be restored in the following ways:

by passing -streamAggr.dropInput command-line flag to single-node VictoriaMetrics;
by passing -remoteWrite.streamAggr.dropInput command-line flag per each configured -remoteWrite.streamAggr.config at vmagent.

SECURITY: upgrade base docker image (alpine) from 3.18.0 to 3.18.2. See alpine 3.18.2 release notes.
SECURITY: upgrade Go builder from Go1.20.5 to Go1.20.6. See the list of issues addressed in Go1.20.6.
FEATURE: reduce memory usage by up to 5x for setups with high churn rate and long retention. See the description for this change and this issue for details.
FEATURE: reduce spikes in CPU and disk IO usage during indexdb rotation (aka inverted index), which is performed once per -retentionPeriod. The new algorithm gradually pre-populates newly created indexdb during the last hour before the rotation. The number of pre-populated series in the newly created indexdb can be monitored via vm_timeseries_precreated_total metric. This should resolve this issue.
FEATURE: MetricsQL: allow selecting time series matching at least one of multiple or filters. For example, {env="prod",job="a" or env="dev",job="b"} selects series with either {env="prod",job="a"} or {env="dev",job="b"} labels. This functionality allows passing the selected series to rollup functions without the need to use subqueries. See these docs.
FEATURE: MetricsQL: add ability to preserve metric names for binary operation results via keep_metric_names modifier. For example, ({__name__=~"foo|bar"} / 10) keep_metric_names leaves foo and bar metric names in division results. See these docs. This helps to address issues like this one.
FEATURE: MetricsQL: add ability to copy all the labels from one side of many-to-one operations by specifying * inside group_left() or group_right(). Also allow adding a prefix for copied label names via group_left(*) prefix "..." syntax. For example, the following query copies Kubernetes namespace labels to kube_pod_info series and adds ns_ prefix for the copied label names: kube_pod_info * on(namespace) group_left(*) prefix "ns_" kube_namespace_labels. The labels from on() list aren't prefixed. This feature resolves this and that questions at StackOverflow.
FEATURE: MetricsQL: add ability to specify durations via WITH templates. Examples:
- WITH (w = 5m) m[w] is automatically transformed to m[5m]
- WITH (f(window, step, off) = m[window:step] offset off) f(5m, 10s, 1h) is automatically transformed to m[5m:10s] offset 1h Thanks to @lujiajing1126 for the initial idea and implementation. See this feature request.
FEATURE: vmui: added a new page with the list of currently running queries. See this issue and these docs.
FEATURE: vmagent: add support for data ingestion via OpenTelemetry protocol. See these docs, this feature request and this pull request.
FEATURE: vmagent: allow sharding outgoing time series among the configured remote storage systems. This can be useful for building horizontally scalable stream aggregation, when samples for the same time series must be aggregated by the same vmagent instance at the second level. See these docs and this feature request for details.
FEATURE: vmagent: allow configuring staleness interval in stream aggregation config. See this issue for details.
FEATURE: vmagent: allow specifying a list of series selectors inside if option of relabeling rules. The corresponding relabeling rule is executed when at least a single series selector matches. See these docs.
FEATURE: stream aggregation: allow specifying a list of series selectors inside match option of stream aggregation configs. The input sample is aggregated when at least a single series selector matches. See this feature request.
FEATURE: stream aggregation: preserve input samples, which match zero match options from the configured aggregations. Previously all the input samples were dropped by default, so only the aggregated samples are written to the output storage. The previous behavior can be restored by passing -streamAggr.dropInput command-line flag to single-node VictoriaMetrics or by passing -remoteWrite.streamAggr.dropInput command-line flag to vmagent.
FEATURE: vmctl: add verbose output for docker installations or when TTY isn't available. See this issue.
FEATURE: vmctl: interrupt backoff retries when import process is cancelled. The change makes vmctl more responsive in case of errors during the import. See this pull request.
FEATURE: vmctl: update backoff policy on retries to reduce probability of overloading for source or destination databases. See this issue.
FEATURE: vmstorage: suppress "broken pipe" and "connection reset by peer" errors for search queries on vmstorage side. See this and this commits.
FEATURE: Official Grafana dashboards for VictoriaMetrics: add panel for tracking rate of syscalls while writing or reading from disk via process_io_(read|write)_syscalls_total metrics.
FEATURE: accept timestamps in milliseconds at start, end and time query args in Prometheus querying API. See these docs and this feature request.
FEATURE: vmalert: update retry policy for pushing data to -remoteWrite.url. By default, vmalert will make multiple retry attempts with exponential delay. The total time spent during retry attempts shouldn't exceed -remoteWrite.retryMaxTime (default is 30s). When retry time is exceeded vmalert drops the data dedicated for -remoteWrite.url. Before, vmalert dropped data after 5 retry attempts with 1s delay between attempts (not configurable). See -remoteWrite.retryMinInterval and -remoteWrite.retryMaxTime cmd-line flags.
FEATURE: vmalert: expose vmalert_remotewrite_send_duration_seconds_total counter, which can be used for determining high saturation of every connection to remote storage with an alerting query sum(rate(vmalert_remotewrite_send_duration_seconds_total[5m])) by(job, instance) > 0.9 * max(vmalert_remotewrite_concurrency) by(job, instance). This query triggers when a connection is saturated by more than 90%. This usually means that -remoteWrite.concurrency command-line flag must be increased in order to increase the number of concurrent writings into remote endpoint. See this feature request.
FEATUTE: vmalert: display the error message received during unsuccessful config reload in vmalert's UI. See this issue for details.
FEATUTE: vmalert: allow disabling of step param attached to instant queries. This might be useful for using vmalert with datasources that to not support this param, unlike VictoriaMetrics. See this issue for details.
FEATUTE: vmalert: support option for "blackholing" alerting notifications if -notifier.blackhole cmd-line flag is set. Enable this flag if you want vmalert to evaluate alerting rules without sending any notifications to external receivers (eg. alertmanager). See this issue for details. Thanks to @venkatbvc for the pull request.
FEATURE: vmalert: add unit test for alerting and recording rules, see more details here. Thanks to @Haleygo for the pull request.
FEATURE: vmalert: allow overriding default GET params for rules with graphite datasource type, in the same way as it happens for prometheus type. See this issue.
FEATUTE: vmalert: support keep_firing_for field for alerting rules. See docs updated here and this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmauth: expose vmauth_user_request_duration_seconds and vmauth_unauthorized_user_request_duration_seconds summary metrics for measuring requests latency per user.
FEATURE: vmbackup: show backup progress percentage in log during backup uploading. See this issue.
FEATURE: vmrestore: show restoring progress percentage in log during backup downloading. See this issue.
FEATURE: add ability to fine-tune Graphite API limits via the following command-line flags: -search.maxGraphiteTagKeys for limiting the number of tag keys returned from Graphite API for tags -search.maxGraphiteTagValues for limiting the number of tag values returned from Graphite API for tag values -search.maxGraphiteSeries for limiting the number of series (aka paths) returned from Graphite API for series See this issue.
BUGFIX: properly return series from /api/v1/series if it finds more than the limit series (limit is an optional query arg passed to this API). Previously the limit exceeded error error was returned in this case. See this issue.
BUGFIX: vmui: fix application routing issues and problems with manual URL changes. See this pull request and this issue.
BUGFIX: add validation for invalid partial RFC3339 timestamp formats in query and export APIs.
BUGFIX: vmctl: interrupt explore procedure in influx mode if vmctl found no numeric fields.
BUGFIX: vmctl: fix panic in case --remote-read-filter-time-start flag is not set for remote-read mode. This flag is now required to use remote-read mode. See this issue.
BUGFIX: vmctl: fix formatting issue, which could add superflouos s characters at the end of samples/s output during data migration. For example, it could write samples/ssssss. See this issue.
BUGFIX: vmalert: use RFC3339 time format in query args instead of unix timestamp for all issued queries to Prometheus-like datasources.
BUGFIX: vmalert: correctly calculate evaluation time for rules. Before, there was a low probability for discrepancy between actual time and rules evaluation time if evaluation interval was lower than the execution time for rules within the group.
BUGFIX: vmalert: reset evaluation timestamp after modifying group interval. Before, there could have latency on rule evaluation time.
BUGFIX: vmselect: fix timestamp alignment for Prometheus querying API if time argument is less than 10m from the beginning of Unix epoch.
BUGFIX: vmagent: close HTTP connections to service discovery servers when they are no longer needed. This should prevent from possible connection exhasution in some cases. See this issue.
BUGFIX: vmagent: do not show relabel debug links at the /targets page when vmagent runs with -promscrape.dropOriginalLabels command-line flag, since it has no the original labels needed for relabel debug. See this issue.
BUGFIX: vminsert: fixed decoding of label values with slash when accepting data via pushgateway protocol. This fixes Prometheus golang client compatibility. See this issue.
BUGFIX: MetricsQL: properly parse binary operations with reserved words on the right side such as foo + (on{bar="baz"}). Previously such queries could lead to panic. See this issue.
BUGFIX: Official Grafana dashboards for VictoriaMetrics: display cache usage for all components on panel Cache usage % by type for cluster dashboard. Before, only vmstorage caches were shown.

v1.91.3

Released at 2023-06-30

SECURITY: upgrade Go builder from Go1.20.4 to Go1.20.5. See the list of issues addressed in Go1.20.5.
BUGFIX: vmagent: fix possible panic at shutdown when stream aggregation is enabled. See this pull request for details.
BUGFIX: vmagent: fixed service name detection for consulagent service discovery in case of a difference in service name and service id. See this issue for details.
BUGFIX: vmalert: retry all errors except 4XX status codes while pushing via remote-write to the remote storage. Previously, errors like broken connection could prevent vmalert from retrying the request.
BUGFIX: vmalert: properly interrupt retry attempts on vmalert shutdown. Before, vmalert could have waited for all retries to finish for shutdown.
BUGFIX: vmbackupmanager: fix an issue with vmbackupmanager not being able to restore data from a backup stored in GCS. See this issue for details.
BUGFIX: VictoriaMetrics cluster: properly return error from /api/v1/query and /api/v1/query_range at vmselect when the -search.maxSamplesPerQuery or -search.maxSamplesPerSeries limit is exceeded. Previously incomplete response could be returned without the error if vmselect runs with -replicationFactor greater than 1. See this pull request.
BUGFIX: storage: prevent from possible crashloop after the migration from versions below v1.90.0 to newer versions. See this issue for details.
BUGFIX: vmui: fix a memory leak issue associated with chart updates. See this pull request.
BUGFIX: vmbackupmanager: fix removing storage data dir before restoring from backup.
BUGFIX: vmselect: wait for all vmstorage nodes to respond when the -replicationFactor flag is set bigger than > 1. Before, vmselect could have skip waiting for the slowest replicas to respond. This could have resulted in issues illustrated here. Now, this optimization is disabled by default and could be re-enabled by passing -search.skipSlowReplicas cmd-line flag to vmselect. See more details here.

v1.91.2

Released at 2023-06-02

BUGFIX: vmalert: fix nil map assignment panic in runtime introduced in this change.

v1.91.1

Released at 2023-06-01

FEATURE:vmagent: Adds follow_redirects at service discovery level of scrape configuration. See this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmselect: Decreases startup time for vmselect with a big number of vmstorage nodes. See this issue. Thanks to @Haleygo for the pull request.
BUGFIX: vmalert: Properly form path to static assets in WEB UI if http.pathPrefix set. See this issue.
BUGFIX: vmalert: Properly set datasource query params. See this issue. Thanks to @gsakun for the pull request.
BUGFIX: vmalert: properly return empty slices instead of nil for /api/v1/rules for groups with present name but absent rules. See this issue.
BUGFIX: vmauth: Properly handle LOCAL command for proxy protocol. See this issue.
BUGFIX: vmbackupmanager: Fixes crash on startup. See this issue.
BUGFIX: vmui: fix bug with custom URL in global settings not respecting tenantID change. See this issue.

v1.91.0

Released at 2023-05-18

SECURITY: upgrade Go builder from Go1.20.3 to Go1.20.4. See the list of issues addressed in Go1.20.4.
SECURITY: serve /robots.txt content to disallow indexing of the exposed instances by search engines. See this issue for details.
FEATURE: update docker compose environment to V2 in respect to V1 deprecation notice from June 2023. See Migrate to Compose V2.
FEATURE: deprecate -bigMergeConcurrency command-line flag, since improper configuration for this flag frequently led to uncontrolled growth of unmerged parts, which, in turn, could lead to queries slowdown and increased CPU usage. The concurrency for background merges can be controlled via -smallMergeConcurrency command-line flag, though it isn't recommended to change this flag in general case.
FEATURE: do not execute the incoming request if it has been canceled by the client before the execution start. See this pull request.
FEATURE: support time formats with timezones. For example, 2024-01-02+02:00 means January 2, 2024 at +02:00 time zone. See these docs.
FEATURE: expose process_* metrics at /metrics page of all the VictoriaMetrics components under Windows OS. See this pull request.
FEATURE: reduce the amounts of unimportant INFO logging during VictoriaMetrics startup / shutdown. This should improve visibility for potentially important logs.
FEATURE: upgrade base docker image (alpine) from 3.17.3 to 3.18.0. See alpine 3.18.0 release notes.
FEATURE: VictoriaMetrics cluster: do not pollute logs with cannot read hello: cannot read message with size 11: EOF messages at vmstorage during TCP health checks performed by Consul or other services. See this issue.
FEATURE: vmagent: support the ability to filter consul_sd_configs targets in more optimal way via new filter option. See this feature request.
FEATURE: vmagent: add support for consulagent_sd_configs. See this feature request.
FEATURE: vmagent: emit a warning if too small value is passed to -remoteWrite.maxDiskUsagePerURL command-line flag. See this issue.
FEATURE: vmalert: add support of recursive globs for -rule and -rule.templates command-line flags by using ** in the glob pattern. See this issue.
FEATURE: vmalert: add ability to specify custom per-group HTTP headers sent to the configured notifiers. See this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmalert: detect alerting rules which don't match any series. See these docs and this feature request.
FEATURE: vmalert: support loading rules via HTTP URL. See this issue. Thanks to @Haleygo for the pull request.
FEATURE: vmalert: add buttons for filtering groups/rules with errors or with no-match warning in web UI for page /groups. See this issue.
FEATURE: vmalert: do not retry remote-write requests for responses with 4XX status codes. This aligns with Prometheus remote write specification. Thanks to @MichaHoffmann for the pull request.
FEATURE: vmauth: add ability to filter incoming requests by IP. See these docs and this feature request.
FEATURE: vmauth: add ability to proxy requests to the specified backends for unauthorized users. See this feature request.
FEATURE: vmauth: add ability to specify default route for unmatched requests. See this feature request.
FEATURE: vmauth: retry POST requests on the remaining backends if the currently selected backend isn't reachable. See this issue.
FEATURE: vmui: add ability to compare the data for the previous day with the data for the current day at Cardinality Explorer. See this feature request.
FEATURE: vmui: display histograms as heatmaps in Metrics explorer. See this feature request.
FEATURE: vmui: add WITH template playground. See this feature request.
FEATURE: vmui: add ability to debug relabeling. See this feature request.
FEATURE: vmui: add an ability to copy and execute queries listed at top queries page. Also make more human readable the query duration column. See this feature request and this pull request.
FEATURE: vmui: increase default font size for better readability.
FEATURE: vmui: cardinality explorer: return back a table with labels containing the highest number of unique label values. See issue.
FEATURE: vmui: add notification icon for queries that do not match any time series. A warning icon appears next to the query field when the executed query does not match any time series. See this feature request.
FEATURE: vmbackup: add -s3StorageClass command-line flag for setting the storage class for AWS S3 backups. See this issue. Thanks to @justcompile for the pull request.
FEATURE: vmbackup: store backup creation and completion time in backup_complete.ignore file of backup contents. This allows determining the exact timestamp when the backup was created and completed.
FEATURE: vmbackupmanager: add created_at field to the output of /api/v1/backups API and vmbackupmanager backup list command. See this doc for data format details.
FEATURE: vmbackupmanager: add commands for locking/unlocking backups against deletion by retention policy. See this doc for data format details.
FEATURE: vmctl: add support for different time formats for --vm-native-filter-time-start and --vm-native-filter-time-end command-line flags. See this issue.
FEATURE: vmctl: set default value for --vm-native-step-interval command-line flag to month. This enables time-based chunking of data based on monthly step value when using native migration mode. See this issue.
BUGFIX: reduce the probability of sudden increase in the number of small parts on systems with small number of CPU cores.
BUGFIX: reduce the possibility of increased CPU usage when data with timestamps older than one hour is ingested into VictoriaMetrics. This reduces spikes for the graph sum(rate(vm_slow_per_day_index_inserts_total)). See this pull request.
BUGFIX: fix possible infinite loop during indexdb rotation when -retentionTimezoneOffset command-line flag is set and the local timezone is not UTC. See this issue. Thanks to @faceair for the fix.
BUGFIX: do not panic at Windows during snapshot deletion. Instead, delete the snapshot on the next restart. See this comment for details.
BUGFIX: change the max allowed value for -memory.allowedPercent from 100 to 200. See this issue.
BUGFIX: properly limit the number of OpenTSDB HTTP concurrent requests specified via -maxConcurrentInserts command-line flag. See this issue. Thanks to @zouxiang1993 for the fix.
BUGFIX: do not ignore trailing empty field in CSV lines when importing data in CSV format. See this issue.
BUGFIX: disallow " chars when parsing Prometheus label names, since they aren't allowed by Prometheus text exposition format. Previously this could result in silent incorrect parsing of incorrect Prometheus labels such as foo{"bar"="baz"} or {foo:"bar",baz="aaa"}. See this issue.
BUGFIX: VictoriaMetrics cluster: prevent from possible panic when the number of vmstorage nodes increases when automatic vmstorage discovery is enabled.
BUGFIX: MetricsQL: fix a panic when the duration in the query contains uppercase M suffix. Such a suffix isn't allowed to use in durations, since it clashes with a million suffix, e.g. it isn't clear whether rate(metric[5M]) means rate over 5 minutes, 5 months or 5 million seconds. See this and this issues.
BUGFIX: vmagent: properly handle the vm_promscrape_config_last_reload_successful metric after config reload. See this issue.
BUGFIX: vmagent: add __meta_kubernetes_endpoints_name label for all ports discovered from endpoint. Previously, ports not matched by Service did not have this label. See this issue for details. Thanks to @thunderbird86 for discovering and fixing the issue.
BUGFIX: vmalert: retry failed read request on the closed connection one more time. This improves rules execution reliability when connection between vmalert and datasource closes unexpectedly.
BUGFIX: vmalert: properly display an error when using query function for templating value of -external.alert.source flag. See this issue.
BUGFIX: vmalert: properly return empty slices instead of nil for /api/v1/rules and /api/v1/alerts API handlers. See this issue.
BUGFIX: vmauth: do not return invalid auth credentials in http response by default, since it may be logged by client. See this issue.
BUGFIX: vmui: fix the display of the tenant selector. See this issue.
BUGFIX: vmui: fix UI freeze when the query returns non-histogram series alongside histogram series.
BUGFIX: vmui: fix the text display on buttons in Safari 16.4.
BUGFIX: alerts-health: update threshold for TooHighMemoryUsage alert from 90% to 80%, since 90% is too high for production environments.
BUGFIX: vmbackup: fix compatibility with Windows OS. See this issue.
BUGFIX: vmctl: fix performance issue when migrating data from VictoriaMetrics according to these docs. Add the ability to speed up the data migration via --vm-native-disable-retries command-line flag. See this issue.
BUGFIX: stream aggregation: fix bug with duplicated labels during stream aggregation via single-node VictoriaMetrics. See this issue.

v1.90.0

Released at 2023-04-06

Update note: this release contains backwards-incompatible change in storage data format, so the previous versions of VictoriaMetrics will exit with the unexpected number of substrings in the part name error when trying to run them on the data created by v1.90.0 or newer versions. The solution is to upgrade to v1.90.0 or newer releases

SECURITY: upgrade base docker image (alpine) from 3.17.2 to 3.17.3. See alpine 3.17.3 release notes.
SECURITY: upgrade Go builder from Go1.20.2 to Go1.20.3. See the list of issues addressed in Go1.20.3.
FEATURE: open source Graphite Render API. This API allows using VictoriaMetrics as a drop-in replacement for Graphite at both data ingestion and querying sides and reducing infrastructure costs by up to 10x comparing to Graphite. See this case study as an example.
FEATURE: release Windows binaries for single-node VictoriaMetrics, VictoriaMetrics cluster, vmbackup and vmrestore. See this, this and this issues. This release of VictoriaMetrics for Windows cannot delete snapshots due to Windows constraints. See this comment for details. This issue should be resolved in future releases.
FEATURE: log metrics with truncated labels if the length of label value in the ingested metric exceeds -maxLabelValueLen. This should simplify debugging for this case.
FEATURE: vmagent: show target URL when debugging target relabeling. This should simplify target relabel debugging a bit. See this pull request.
FEATURE: vmagent: add support for VictoriaMetrics remote write protocol when sending / receiving data to / from Kafka. This protocol allows saving egress network bandwidth costs when sending data from vmagent to Kafka located in another datacenter or availability zone. See this feature request.
FEATURE: vmagent: add -kafka.consumer.topic.concurrency command-line flag. It controls the number of Kafka consumer workers to use by vmagent. It should eliminate the need to start multiple vmagent instances to improve data transfer rate. See this feature request.
FEATURE: vmagent: add support for Kafka producer and consumer on arm64 machines. See this issue.
FEATURE: vmagent: delete unused buffered data at -remoteWrite.tmpDataPath directory when there is no matching -remoteWrite.url to send this data to. See this feature request.
FEATURE: vmagent: add the ability for hot reloading of stream aggregation configs. See these docs and this feature request.
FEATURE: check the contents of -relabelConfig and -streamAggr.config files additionally to -promscrape.config when single-node VictoriaMetrics runs with -dryRun command-line flag. This aligns the behaviour of single-node VictoriaMetrics with vmagent behaviour for -dryRun command-line flag.
FEATURE: vmui: automatically draw a heatmap graph when the query selects a single histogram. This simplifies analyzing histograms. See this feature request.
FEATURE: vmui: add support for drag'n'drop and paste from clipboard in the "Trace analyzer" page. See this pull request.
FEATURE: vmui: hide messages longer than 3 lines in the trace. You can view the full message by clicking on the show more button. See this pull request.
FEATURE: vmui: add the ability to manually input date and time when selecting a time range. See this pull request.
FEATURE: vmui: updated usability and the search process in cardinality explorer. Made this process straightforward for user. See this pull request.
FEATURE: vmui: add the ability to collapse/expand the legend. See this pull request.
FEATURE: vmui: add tips for working with the graph and legend. See this pull request.
FEATURE: vmui: add apply and cancel buttons to settings popup. See this issue.
FEATURE: vmctl: automatically disable progress bar when TTY isn't available. See this issue.
FEATURE: vmauth: add -configCheckInterval command-line flag, which can be used for automatic re-reading the -auth.config file. See this feature request.
BUGFIX: prevent from slow snapshot creating under high data ingestion rate. See this issue.
BUGFIX: vmauth: suppress proxy protocol parsing errors in case of EOF. Usually, the error is caused by health checks and is not a sign of an actual error.
BUGFIX: vmui: fix displaying errors for each query. See this issue.
BUGFIX: vmbackup: fix snapshot not being deleted in case of error during backup. See this issue.
BUGFIX: stream aggregation: suppress series after dedup error message in logs when -remoteWrite.streamAggr.dedupInterval command-line flag is set at vmagent or when -streamAggr.dedupInterval command-line flag is set at single-node VictoriaMetrics.
BUGFIX: allow using dashes and dots in environment variables names referred in config files via %{ENV-VAR.SYNTAX}. See these docs and this issue.
BUGFIX: return back query performance scalability on hosts with big number of CPU cores. The scalability has been reduced in v1.86.0. See this issue.
BUGFIX: MetricsQL: properly convert VictoriaMetrics historgram buckets to Prometheus histogram buckets when VictoriaMetrics histogram contain zero buckets. Previously these buckets were ignored, and this could lead to missing Prometheus histogram buckets after the conversion. Thanks to @zklapow for the fix.
BUGFIX: vmagent: fix CPU and memory usage spikes when files pointed by file_sd_config cannot be re-read. See this_issue.
BUGFIX: prevent unexpected merges on start-up when -storage.minFreeDiskSpaceBytes is set. See the issue.
BUGFIX: properly support comma-separated filters inside retention filters. See this issue.
BUGFIX: verify response code when fetching configuration files via HTTP. See this issue.
BUGFIX: vmalert: replace empty labels with "" instead of "<no value>" during templating, as Prometheus does. See this issue.
BUGFIX: vmctl: properly pass multiple filters from --vm-native-filter-match command-line flag to the data source. Previously filters from --vm-native-filter-match were only used to discover the metric names, and the metric names like __name__="metric_name" has been taken into account, while the remaining filters were ignored. For example --vm-native-src-addr={foo="bar",baz="abc"} may found metric_name{foo="bar",baz="abc"} and filter was treated as --vm-native-src-addr={__name__="metrics_name"}, e.g. foo="bar",baz="abc" filter was ignored. See this issue.

v1.89.1

Released at 2023-03-12

BUGFIX: prevent from possible cannot unmarshal timeseries from rollupResultCache panic after the upgrade to v1.89.0.

v1.89.0

Released at 2023-03-12

Update note: this release can crash with cannot unmarshal timeseries from rollupResultCache panic after the upgrade from the previous releases. This issue can be fixed by removing caches stored on disk according to these docs. Another option is to upgrade to v1.89.1.

SECURITY: upgrade Go builder from Go1.20.1 to Go1.20.2. See the list of issues addressed in Go1.20.2.
FEATURE: vmctl: increase the default value for --remote-read-http-timeout command-line option from 30s (30 seconds) to 5m (5 minutes). This reduces the probability of timeout errors when migrating big number of time series. See this pull request.
FEATURE: vmctl: migrate series one-by-one in vm-native mode. This allows better tracking the migration progress and resuming the migration process from the last migrated time series. See this pull request and this feature request.
FEATURE: vmctl: add --vm-native-src-headers and --vm-native-dst-headers command-line flags, which can be used for setting custom HTTP headers during vm-native migration mode. Thanks to @baconmania for the pull request.
FEATURE: vmctl: add --vm-native-src-bearer-token and --vm-native-dst-bearer-token command-line flags, which can be used for setting Bearer token headers for the source and the destination storage during vm-native migration mode. See this feature request.
FEATURE: vmctl: add --vm-native-disable-http-keep-alive command-line flag to allow vmctl to use non-persistent HTTP connections in vm-native migration mode. Thanks to @baconmania for the pull request.
FEATURE: vmalert: log number of configration files found for each specified -rule command-line flag.
FEATURE: vmalert enterprise: concurrently read config files from S3, GCS or S3-compatible object storage. This significantly improves config load speed for cases when there are thousands of files to read from the object storage.
BUGFIX: vmstorage: fix a bug, which could lead to incomplete or empty results for heavy queries selecting tens of thousands of time series. See this pull request.
BUGFIX: vmselect: reduce memory usage and CPU usage when performing heavy queries. See this issue.
BUGFIX: prevent from possible invalid memory address or nil pointer dereference panic during background merge. The issue has been introduced at v1.85.0. See this issue.
BUGFIX: prevent from possible SIGBUS crash on ARM architectures (Raspberry Pi), which deny unaligned access to 8-byte words. Thanks to @oliverpool for narrowing down the issue and for the initial attempt to fix it.
BUGFIX: VictoriaMetrics cluster: always return is_partial: true in partial responses. Previously partial responses could be returned as non-partial in some cases.
BUGFIX: VictoriaMetrics cluster: properly take into account -rpc.disableCompression command-line flag at vmstorage. It was ignored since v1.78.0. See this pull request.
BUGFIX: vmagent: fix panic when writing data to Kafka. The panic has been introduced in v1.88.0.
BUGFIX: vmui: stop showing Please enter a valid Query and execute it error message on the first load of vmui.
BUGFIX: vmui: properly process Run in VMUI button click in VictoriaMetrics datasource plugin for Grafana.
BUGFIX: vmui: fix the display of the selected value for dropdowns on Explore page.
BUGFIX: vmui: do not send step param for instant queries. See this issue.
BUGFIX: vmauth: fix cannot serve http panic when plain HTTP request is sent to vmauth configured to accept requests over proxy protocol-encoded request (e.g. when vmauth runs with -httpListenAddr.useProxyProtocol command-line flag). The issue has been introduced at v1.87.0 when implementing this feature.
BUGFIX: vmgateway: properly parse RSA public key discovered via JWK endpoint.

v1.88.1

Released at 2023-02-27

FEATURE: add -snapshotCreateTimeout flag to allow configuring timeout for snapshot process. See this issue.
FEATURE: expose vm_http_requests_total and vm_http_request_errors_total metrics for snapshot/* paths at VictoriaMetrics cluster vmstorage and VictoriaMetrics Single. See this issue.
FEATURE: vmgateway: add the ability to discover keys for JWT verification via OpenID discovery endpoint. See these docs.
FEATURE: add -internStringDisableCache command-line flag for disabling the cache for interned strings. This flag may be useful in some cases for reducing memory usage at the cost of higher CPU usage.
FEATURE: add -internStringCacheExpireDuration command-line flag for controlling the lifetime of cached interned strings.
BUGFIX: MetricsQL: fix panic when executing the query aggr_func(rollup*(some_value)). The panic has been introduced in v1.88.0.
BUGFIX: vmagent: use the provided -remoteWrite.* auth options when determining whether the remote storage supports VictoriaMetrics remote write protocol. Previously the auth options were ignored. This was preventing from automatic switch to VictoriaMetrics remote write protocol.
BUGFIX: vmagent: do not register vm_promscrape_config_* metrics if -promscrape.config flag is not used. Previously those metrics were registered and never updated, which was confusing and could trigger false-positive alerts.
BUGFIX: vmctl: skip measurements with no fields when migrating data from influxdb. See this issue.
BUGFIX: delete failed snapshot contents from disk on failed attempt to create snapshot. Previously failed snapshot contents could remain on disk in incomplete state. See this issue

v1.88.0

Released at 2023-02-24

SECURITY: upgrade base docker image (alpine) from 3.17.1 to 3.17.2. See alpine 3.17.2 release notes.
SECURITY: upgrade Go builder from Go1.20.0 to Go1.20.1. See the list of issues addressed in Go1.20.1.
FEATURE: vmagent: add support for VictoriaMetrics remote write protocol. This protocol allows saving egress network bandwidth costs when sending data from vmagent to VictoriaMetrics located in another datacenter or availability zone. This also allows reducing disk IO under high load when vmagent starts queuing the collected data to disk when the remote storage is temporarily unavailable or cannot keep up with the data ingestion rate. See this feature request.
FEATURE: vmagent: add support for Kuma Control Plane targets discovery aka kuma_sd_configs. See this issue.
FEATURE: vmgateway: add the ability to verify JWT signature via JWKS endpoint. See these docs.
FEATURE: vmauth: add the ability to limit the number of concurrent requests on a per-user basis via -maxConcurrentPerUserRequests command-line flag and via max_concurrent_requests config option. See this feature request and these docs.
FEATURE: vmauth: automatically retry failing GET requests on all the configured backends. Previously the backend error has been immediately returned to the client without retrying the request on the remaining backends.
FEATURE: vmauth: choose the backend with the minimum number of concurrently executed requests among the configured backends in a round-robin manner for serving the incoming requests. This allows spreading the load among backends more evenly, while improving the response time.
FEATURE: vmalert enterprise: add ability to read alerting and recording rules from S3, GCS or S3-compatible object storage. See these docs.
FEATURE: vmctl: automatically retry requests to remote storage if up to 5 errors occur during the data migration process. This should help continuing the data migration process on temporary errors. Previously vmctl was stopping after the first error. See this feature request.
FEATURE: MetricsQL: support optional 2nd argument min, max or avg for rollup, rollup_delta, rollup_deriv, rollup_increase, rollup_rate and rollup_scrape_interval function. If the second argument is passed, then the function returns only the selected aggregation type. This change can be useful for situations where only one type of rollup calculation is needed. For example, rollup_rate(requests_total[1i], "max") would return only the max increase rates for requests_total metric per each interval between adjacent points on the graph. See this article for details.
FEATURE: MetricsQL: support optional 2nd argument open, low, high, close for rollup_candlestick function. If the second argument is passed, then the function returns only the selected aggregation type.
FEATURE: MetricsQL: add share(q) aggregate function.
FEATURE: MetricsQL: add mad_over_time(m[d]) function for calculating the median absolute deviation over raw samples on the lookbehind window d. See this feature request.
FEATURE: MetricsQL: add range_mad(q) function for calculating the median absolute deviation over points per each time series returned by q.
FEATURE: MetricsQL: add range_zscore(q) function for calculating z-score over points per each time series returned from q.
FEATURE: MetricsQL: add range_trim_outliers(k, q) function for dropping outliers located farther than k*range_mad(q) from the range_median(q). This should help removing outliers during query time at this issue.
FEATURE: MetricsQL: add range_trim_zscore(z, q) function for dropping outliers located farther than z*range_stddev(q) from range_avg(q). This should help removing outliers during query time at this issue.
FEATURE: vmui: show median instead of avg in graph tooltip and line legend, since median is more tolerant against spikes. See this issue.
FEATURE: add -search.maxSeriesPerAggrFunc command-line flag, which can be used for limiting the number of time series MetricsQL aggregate functions can return in a single query. This flag can be useful for preventing OOMs when count_values function is improperly used.
FEATURE: vmui: small UX improvements for mobile view. See this feature request and this pull request.
FEATURE: add -search.logQueryMemoryUsage command-line flag for logging queries, which need more memory than specified by this command-line flag. See this feature request. Thanks to @michal-kralik for the idea and the intial implementation.
FEATURE: allow setting zero value for -search.latencyOffset command-line flag. This may be needed in some cases. Previously the minimum supported value for -search.latencyOffset command-line flag was 1s.
BUGFIX: vmagent: immediately cancel in-flight scrape requests during configuration reload when stream parsing mode is disabled. Previously vmagent could wait for long time until all the in-flight requests are completed before reloading the configuration. This could significantly slow down configuration reload. See this issue.
BUGFIX: vmagent: do not wait for 2 seconds after the first unsuccessful attempt to scrape the target before performing the next attempt. This should improve scrape speed when the target closes http keep-alive connection between scrapes. See this and this issues.
BUGFIX: vmagent: fix Azure service discovery inside Azure Container App. See this issue. Thanks to @MattiasAng for the fix!
BUGFIX: do not put auxiliary directories scheduled for removal into snapshots. This should prevent from cannot create hard links from ...must-remove... errors when making snapshots / backups. See this issue.
BUGFIX: prevent from possible data ingestion slowdown and query performance slowdown during background merges of big parts on systems with small number of CPU cores (1 or 2 CPU cores). The issue has been introduced in v1.85.0 when implementing this feature. See also this issue.
BUGFIX: properly parse timestamps in milliseconds when ingesting data via OpenTSDB telnet put protocol. Previously timestamps in milliseconds were mistakenly multiplied by 1000. Thanks to @Droxenator for the pull request.
BUGFIX: MetricsQL: do not add extrapolated points outside the real points when using interpolate() function. See this issue.