Reduce Prometheus Resource Use with Metric Filtering

Reduce Prometheus resource use by controlling the number of targets, active time series, label cardinality, scrape frequency and retention. Metric relabeling can discard samples before storage, but Prometheus still fetches and parses the scrape response. Measure the result against the workload and alerts you need to keep.

Test a minimal configuration

This local Docker example runs Prometheus 3.15.1 and scrapes its own metrics. Save the following as prometheus.yml.

global:
  scrape_interval: 5s
  scrape_timeout: 4s
scrape_configs:
  - job_name: self
    static_configs:
      - targets: ['localhost:9090']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: process_cpu_seconds_total
        action: keep

docker run --rm --entrypoint /bin/promtool \
  -v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
  prom/prometheus:v3.15.1 check config /etc/prometheus/prometheus.yml
docker run -d --name cloudhero-prometheus-lab \
  -p 127.0.0.1:19090:9090 \
  -v "$PWD/prometheus.yml:/etc/prometheus/prometheus.yml:ro" \
  prom/prometheus:v3.15.1 \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.retention.time=1h
# After at least two successful scrapes:
curl -fsSG http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=process_cpu_seconds_total{job="self"}'
curl -fsSG http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=go_goroutines{job="self"}'

The CPU series should exist and the excluded Go metric should have no result. Scrape-generated series such as up can remain. A successful config check establishes syntax, not that every production scrape target is reachable.

Apply the idea to Kubernetes carefully

Use authenticated discovery and a trusted CA for API-server or kubelet access. Avoid copying insecure_skip_verify: true. Node proxy permission is sensitive; select the narrowest supported metrics access for your environment. Prometheus Operator configurations may require ServiceMonitor or PodMonitor resources.

Measure the tradeoff

Compare active series, ingestion rate, resident memory, CPU and disk over the same traffic period. Filtering labels can accidentally merge distinct series; dropping metrics can break rules and HPA inputs. Validate those consumers before reducing collection. The original article’s memory observations described one historical installation and are not a general savings estimate.

Remove the lab

docker rm -f cloudhero-prometheus-lab

Sources and further reading

• Prometheus configuration
• Prometheus storage

Check scrape timing and protocol compatibility

A scrape timeout cannot exceed its interval. Our live Helm test initially retained the chart’s explicit ten-second timeout while reducing the interval to five seconds; Prometheus rejected that configuration. Setting a four-second timeout corrected it. A test exporter also needed a valid metrics Content-Type for Prometheus 3. Inspect target errors and actual samples before assuming a chart installation means collection is working.