### What did you do? used following config snippet to set LB service external IP as param_target for DNS probes ``` .... relabel_configs: - action: keep regex: powerdns-external;pdns-udp source_labels: - __meta_kubernetes_service_name - __meta_kubernetes_service_port_name - source_labels: - __meta_kubernetes_service_loadbalancer_ip target_label: __param_targ
importance 15@whitepiratebakuopenhelp wantedcomponent/service discoverygood first issuecomponent/service discovery/kubernetesOriginal evidence ↗
**Current implementation of the RW2 sending does not implement exemplars correctly.** It keeps the RW1 way of exemplars, where they are sent in a separate `TimeSeries` message with series labels, exemplar details and without samples. This represents the way Prometheus V1 Appender looks like and the current exemplar in-mem (and WAL) storage. We could call thi
### Proposal Is't possible to use information about metric in alert rule, but it's not possible tu use activeAt information. It will be great to have access to it. The question have already been asked #14713, but the solution of going through the alertmanager is only a palliative and does not meet all needs. is it possible to modify following files to add "a
### What did you do? If you specify scraping annotation for a kubernetes pod, typically for each container port a target is added (kubernetes/pod.go): https://github.com/prometheus/prometheus/blob/main/discovery/kubernetes/pod.go#L280 If you have a second container in the pod, that does not expose any container ports, then kubernetes/pod.go will add the IP a
This issue lists Renovate updates and detected dependencies. Read the [Dependency Dashboard](https://docs.renovatebot.com/key-concepts/dashboard/) docs to learn more.<br>[View this repository on the Mend.io Web Portal](https://developer.mend.io/github/prometheus/prometheus). ## Repository Problems These problems occurred while renovating this repository. [Vi
Note: I've generated this issue with Claude Opus 4.7 after using it to verify the issue. While reviewing #18768 I noticed that `(N)` inside a duration-expression position (offset, range bracket) is dropped on `Expr.String()` round-trip when N reduces to a single `*NumberLiteral`. Wrapping a multi-term `*DurationExpr` survives because `*DurationExpr` has a `W
#### Which issue(s) does the PR fix: Fixes #19273 #### What this PR does: `clamp()` documents that float samples come back as `NaN` when either bound is `NaN`: > * Float samples are clamped to `NaN` if `min` or `max` is `NaN` `clamp_max()` and `clamp_min()` route through the same `clamp()` helper in `promql/functions.go`, so they behave the same way, but nei
<!-- - Please give your PR a title in the form "area: short description". For example "tsdb: reduce disk usage by 95%" - Please sign CNCF's Developer Certificate of Origin and sign-off your commits by adding the -s / --signoff flag to `git commit`. See https://github.com/apps/dco for more information. - If the PR adds or changes a behaviour or fixes a bug of
There are a couple of links in the documentation where a space exists between the link text ([]) and the URL (()). This breaks standard Markdown link formatting and prevents the links from rendering correctly. The affected links are: * AirBnB's Nerve in https://prometheus.io/docs/prometheus/latest/configuration/configuration/#nerve_sd_config * Serversets in
Fixes #14398 #### Does this PR introduce a user-facing change? <!-- If no, just write "NONE" in the release-notes block below. Otherwise, please describe what should be mentioned in the CHANGELOG. Use the following prefixes: [FEATURE] [ENHANCEMENT] [PERF] [BUGFIX] [SECURITY] [CHANGE] Refer to the existing CHANGELOG for inspiration: https://github.com/prometh
### Problem The PromQL parser allowed invalid syntax like: ```promql metric offset 5m[2m:10s] # Should error, but was accepted metric @ 123[2m:10s] # Should error, but was accepted ``` While correctly rejecting the same pattern for regular range selectors: ```promql metric offset 5m[2m] # Correctly rejected ``` ### Solution Inline a presence check in both `s
<!-- - Please give your PR a title in the form "area: short description". For example "tsdb: reduce disk usage by 95%" - Please sign CNCF's Developer Certificate of Origin and sign-off your commits by adding the -s / --signoff flag to `git commit`. See https://github.com/apps/dco for more information. - If the PR adds or changes a behaviour or fixes a bug of
Fixes #19206 **What this PR does / why we need it**: This PR fixes an issue where encountering a division or modulo by zero (or an out of range) inside a duration expression correctly evaluates to a dummy `NumberLiteral{Val: 0}` to raise an error, but was failing to capture the `PosRange` of the evaluated expression. Because it lacked a positional range, it
Fixes #19263 This PR promotes `retry_on_http_429` in `remote_write` configuration to GA while keeping default = false (leaving the default behavior change for a separate discussion/PR per maintainer feedback). ### Details - Removed experimental note for `retry_on_http_429` in documentation docs/configuration/configuration.md. - Maintained RetryOnRateLimit de
Fixes #17857 This PR ensures that samples, exemplars, and histograms belonging to the same series are grouped into the same `TimeSeries` message within a single Prometheus Remote Write v2 request. Previously, these were sent as separate `TimeSeries` messages, which could result in them being split across different batches or flush boundaries. ```release-note
Add a way to capture the original hash of labels that is "out of bound" of the labels model. This is because even with the best encoding schemes like base 91 storing a hash is wasteful. Also, comparing strings is much slower than comparing uint64s. #### Which issue(s) does the PR fix: NA #### Release notes for end users (**ALL** commits must be considered).
Allows TestDocumentation to run on windows. The two issues preventing this before were; 1. The test binary name differs between platforms. On non-Windows platforms it is _**prometheus.test**_, while on Windows it is _**prometheus.test.exe**_. Added a platform-specific check to handle the .exe suffix. 2. The loaded _expectedContext_ contained Windows-style li
TL;DR; I saw TestRuntimeGOGCConfig fail in a CI. At first I suspected that it cannot be run in parallel due to environment use, but it's only setting sub process env, so that wasn't it. The issue is that it's not waiting long enough for the process, which can take it's time in CI. LLM also noticed another bug with developer settings leaking into the test. Bo
<!-- - Please give your PR a title in the form "area: short description". For example "tsdb: reduce disk usage by 95%" - Please sign CNCF's Developer Certificate of Origin and sign-off your commits by adding the -s / --signoff flag to `git commit`. See https://github.com/apps/dco for more information. - If the PR adds or changes a behaviour or fixes a bug of
This PR extends the STACKIT service discovery (`stackit_sd_configs`) to support discovering STACKIT Postgres Flex instances alongside standard IaaS servers. This PR pretends to add more features started here https://github.com/prometheus/prometheus/pull/16401 . It introduces a new `role` configuration option (`server`, `postgres`, `all`, defaulting to `all`)
#### Which issue(s) does the PR fix: There is no dedicated tracking issue for this bug. (The only open Lightsail issue, #17395, is a request to add unit tests, not this pagination bug.) `GetInstances` is a paginated Lightsail API, but the Lightsail client adapter called it exactly once and never followed the returned `NextPageToken`, so on any Lightsail acco
#### Which issue(s) does the PR fix: Ref: https://github.com/grafana/mimir/issues/4541 #### Release notes for end users (**ALL** commits must be considered). *Reviewers should verify clarity and quality.* ```release-notes [BUGFIX] TSDB: Fix out-of-order chunk ID overflow by wrapping `firstOOOChunkID` modulo 2^23 instead of growing unbounded. ``` ## Summary F
Follow up on #19325. Staleness is tracked in `scrapeCache.seriesCur`/`seriesPrev`, both keyed by `storage.SeriesRef`. When the storage hands out a new reference for a series it already gave one for (e.g. it was garbage collected and had to be recreated), the scrape loop updates the cached reference and tracks the current scrape under the new one, while the p
The old test forced a chunk write failure by removing the **_mmapped_** chunk directory which will not work on Windows, as directories with open files cannot be deleted. This change instead injects a rogue chunk segment with a higher sequence number, causing the next chunk write to fail through the same code path in an OS-independent way. This allows the tes
PR addresses two issues in within `config/config_test.go` and `config/config_windows_test.go` in two separate commits. 1. `DefaultOTLPConfig` was missing from the `ruleFilesExpectedConf` which caused `TestLoadConfigRuleFilesAbsolutePath` to fail on **Windows**. 2. `filepath.FromSlash` is used extensively to handle platform pathing differences, but was missin
## Changes Adds a Troubleshooting section covering: - Common scraping issues and fixes - High memory usage solutions - Alert rule debugging - Useful links to official docs This helps new users debug common problems without searching through issues.
## Summary Fixes #19374 — EC2 service discovery panics when `DescribeInstances` returns instances with a nil `Placement` field (or nil `ImageId`). `discovery/aws/ec2.go` refresh() dereferences `inst.Placement.AvailabilityZone` and `inst.ImageId` with no nil guard. When either is nil, this panics and crashes the entire Prometheus process (there is no `recover
<!-- - Please give your PR a title in the form "area: short description". For example "tsdb: reduce disk usage by 95%" - Please sign CNCF's Developer Certificate of Origin and sign-off your commits by adding the -s / --signoff flag to `git commit`. See https://github.com/apps/dco for more information. - If the PR adds or changes a behaviour or fixes a bug of
ECS SD dereferences optional task fields without guarding them. Any task missing one panics the refresh goroutine and takes the process down instead of degrading a single target. `types.Task.Group`, `TaskArn`, `LastStatus`, `DesiredStatus`, `AvailabilityZone`, `ClusterArn`, `ClusterName` and `TaskDefinitionArn` are all optional — none carry the SDK's `This m
Unix socket scrape support passes the socket path to a shared HTTP client's `DialContext` via the request context. But the transport keys idle connections on scheme and `host:port` only, so two targets sharing the same `__address__` with different `__scrape_unix_socket__` paths can reuse each other's pooled connections and scrape the wrong endpoint. The same
Follow-up to #19216: the HeadChunkID doc still promised a monotonically increasing per-series number, which no longer holds for out-of-order chunk IDs now that they wrap modulo 2^23. Also spell out the "much less than" relation in the oooHeadChunkID comment, where << is easy to misread as a bit-shift. <!-- - Please give your PR a title in the form "area: sho
#### Which issue(s) does the PR fix: Fixes #18770 #### Release notes for end users (**ALL** commits must be considered). *Reviewers should verify clarity and quality.* ```release-notes [BUGFIX] PromQL: Preserve parentheses around duration literals on Expr.String() round-trip. ``` `paren_duration_expr` only set `Wrapped` on `*DurationExpr`, so a parenthesised
#### Which issue(s) does the PR fix: Fixes #14398 #### Release notes for end users (**ALL** commits must be considered). *Reviewers should verify clarity and quality.* ```release-notes [BUGFIX] discovery/kubernetes: Populate `__meta_kubernetes_service_loadbalancer_ip` from `status.loadBalancer.ingress`, falling back to deprecated `spec.loadBalancerIP`. ``` K
#### Which issue(s) does the PR fix: Follow-up to #19396 (same nil-deref class in AWS SD). #### Release notes for end users (**ALL** commits must be considered). *Reviewers should verify clarity and quality.* ```release-notes [BUGFIX] discovery/aws: Don't panic on ElastiCache caches or clusters with absent optional fields. ``` `addServerlessCacheTargets` / `
`matrix_selector` already rejects `foo offset 5m[5m]` and `foo @ 1234[5m]`, but it only looked at `OriginalOffset` and `Timestamp`. Duration-expression offsets and `@ start()` / `@ end()` live in `OriginalOffsetExpr` and `StartOrEnd`, so these still parsed: ```promql foo offset step()[5m] foo @ start()[5m] foo @ end()[5m] ``` The valid forms remain `foo[5m]
#18259 stopped tests from truncating SD files to zero bytes, but `requireUpdate` still treated the first snapshot with the expected *number* of groups as the result. File SD can emit empty groups (`[{"targets":[]},{"targets":[]}]`) before the file contents are visible, so `TestInvalidFileUpdate` and `TestUpdateFileWithPartialWrites` still flake. Wait until t
The generic relabel docs say `__address__` is always `<host>:<port>`. Kubernetes pod SD sets `__address__` to the pod IP with no port when a container declares none, so a port can be added via relabeling. That is already described under the `pod` role; this makes the relabel section match. cc @brancz @rexagod @machine424 Fixes #11678 #### Which issue(s) does
Bounds the reusable TSDB record-buffer pool at 1 MB so unusually large OOO/WBL batches can be collected after commit. Buffers at or below the limit remain reusable. Adds a regression test for the retention limit and a benchmark covering both sides of the threshold. In six benchmark runs, the 1 MB case stayed at about 29.5 ns/op with zero allocations. For the
I was trying to use tracing to debug some rules skipping evaluation and other random slowness but I find current set of spans not really giving me enough information. This adds a bunch more spans all over the place: - engine now shows a span per series selected from tsdb, so a query that's doing a ton of selects shows them all more clearly - rule evaluations
This helps us better understand how many appenders are created per second. There is already prometheus_tsdb_head_active_appenders but it's a gauge, so it only shows active ones. There's also prometheus_tsdb_head_samples_appended_total, but that counts samples. This new counter adds a different view - how many appenders are created per second, combined with p
Range-query and compaction readers cache collected head chunks for sequential access. When a reader moved from a series with more than `headChunksBufMaxCap` head chunks to a different series with zero or one head chunk, the direct-lookup fast path returned before applying the capacity policy. The reader then retained the oversized backing array and its chunk
Previously, the info function did not properly implement delayed name removal, and did not propagate the `dropName` flag properly. This PR makes the info function behave more consistently with other functions like `label_replace` with respect to delayed name removal. #### Which issue(s) does the PR fix: N/A Problem found in https://github.com/prometheus/prom
<!-- - Please give your PR a title in the form "area: short description". For example "tsdb: reduce disk usage by 95%" - Please sign CNCF's Developer Certificate of Origin and sign-off your commits by adding the -s / --signoff flag to `git commit`. See https://github.com/apps/dco for more information. - If the PR adds or changes a behaviour or fixes a bug of
#### Which issue(s) does the PR fix: This PR adds [Zizmor](https://github.com/zizmorcore/zizmor) for GitHub Actions static analysis and also add the workflow to the repo_sync automation, to ensure the workflow is added to all repositories in the Prometheus and Prometheus-community organizations. #### Release notes for end users (**ALL** commits must be consi