Kubernetes Autoscaling with HPA and Prometheus: What to Validate

Kubernetes Horizontal Pod Autoscaling changes replica counts; node autoscaling provides capacity for those replicas. CPU scaling uses the resource metrics API, while Prometheus-based custom scaling needs an adapter that exposes the required metrics API. Validate each layer before interpreting an HPA target.

Start with a stable HPA object

This API example targets a Deployment called web in a disposable namespace. It requires that Deployment, CPU requests on its containers and a working resource metrics API. Save as hpa.yaml.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web
  namespace: autoscale-lab
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 1
  maxReplicas: 5
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60

kubectl create namespace autoscale-lab
kubectl apply --dry-run=server -f hpa.yaml
kubectl get --raw /apis/metrics.k8s.io/v1beta1
# Apply only once the target Deployment and metrics API are ready:
# kubectl apply -f hpa.yaml
# kubectl -n autoscale-lab describe hpa web

A successful dry-run checks the API shape. It does not prove that metrics are available or that a load test scales correctly. CPU utilization is relative to requests; it is not a rule that always adds exactly one Pod above the threshold.

Expose a Prometheus metric through an adapter

For custom metrics, define an adapter rule matching real metric names, labels and units from your installed exporters. Verify the corresponding custom.metrics.k8s.io response for every selected Pod and namespace. Current labels often differ from older examples using pod_name. Do not copy a historical PromQL expression without querying its output.

Lessons from our Magento deployment

Response time looked attractive as a scaling signal, but database or Redis overload could make requests slower without making application CPU busy. Adding PHP workers could then increase pressure on the dependency. Smoothing CPU over a longer window helped with spikes in that installation, while also delaying response to growth. Those observations describe that workload, not a universal best metric.

Validate scale-out, scale-in and failure

1. Establish baseline resource requests, latency and error rate.
2. Increase a representative load and observe HPA conditions and replicas.
3. Confirm Pending Pods can obtain capacity within scheduling and quota limits.
4. Reduce load and inspect stabilization and disruption.
5. Simulate an overloaded dependency and confirm scaling does not worsen it.

Prometheus scrape annotations work only when discovery honors them. Operator installations may use ServiceMonitor or PodMonitor resources. A node autoscaler such as Karpenter has a separate lifecycle and budget.

Clean up the API lab

kubectl delete namespace autoscale-lab

The CPU example above was API-validated. A separate live custom-metric run on EKS exercised Prometheus discovery, the adapter API, scale-out under load, scale-in, and a failed dependency. It used a successful-request rate, not the historical Magento exporter queries. Read Prometheus filtering before dropping metrics used by an HPA.

Sources and further reading

• Kubernetes HPA
• Prometheus Adapter configuration

What the live custom-metric test verified

Each test Pod exposed successful_requests_total with its namespace and Pod name. Prometheus scraped it every five seconds with a four-second timeout. The exporter returned Content-Type: text/plain; version=0.0.4; charset=utf-8; Prometheus 3 rejected the original test endpoint when that header was missing.

This adapter rule translated the counter to a per-Pod rate. Use it only with an exporter that actually exposes these names and labels.

rules:
  default: false
  custom:
    - seriesQuery: 'successful_requests_total{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: namespace}
          pod: {resource: pod}
      name:
        matches: successful_requests_total
        as: successful_requests_per_second
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[30s])) by (<<.GroupBy>>)'

The live HPA used a Pods metric, a target of five successful requests per second per Pod, a minimum of one and maximum of four replicas, and a short scale-down window for the test. Traffic increased replicas; after traffic stopped the deployment returned to one. Failed dependency responses did not increase the successful-work metric. That result validates this policy and fixture; it is not proof that the same metric fits another application.

metrics:
  - type: Pods
    pods:
      metric:
        name: successful_requests_per_second
      target:
        type: AverageValue
        averageValue: '5'