Kubernetes Logging with Fluent Bit, Elasticsearch and Kibana

A Kubernetes logging pipeline usually collects container stdout and stderr from nodes, enriches records with workload metadata and sends them to a searchable backend. Fluent Bit and Fluentd are separate projects. Elasticsearch stores and searches records; Kibana explores the stored data.

Keep application logs on stdout and stderr

Use a node collector, commonly a DaemonSet, to follow container logs. Redirecting application output into a shared file can remove useful kubectl logs visibility and introduce rotation and permission problems. Sidecars remain useful for applications that cannot emit suitable streams, but need their own resource and lifecycle design.

Parse the runtime envelope first

Modern Kubernetes runtimes commonly write CRI-format logs. Configure the tail input’s supported Docker/CRI multiline handling before parsing application content. A Java stack trace is a separate multiline problem from reconstructing split container records. Do not combine the old legacy multiline options with the newer multiline-core settings.

Validate a small Fluent Bit configuration

Save this local smoke-test configuration as fluent-bit.conf. It verifies one record passes through Fluent Bit 5.1.5; it is not a node collector or a complete EFK installation.

[SERVICE]
    Flush 1
    Log_Level info
[INPUT]
    Name dummy
    Tag lab.logs
    Dummy {"message":"logging-lab","kind":"access"}
    Samples 1
[OUTPUT]
    Name stdout
    Match lab.logs

docker run --rm \
  -v "$PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro" \
  fluent/fluent-bit:5.1.5 --dry-run \
  -c /fluent-bit/etc/fluent-bit.conf
# Run briefly, confirm the logging-lab record, then stop with Ctrl+C:
docker run --rm \
  -v "$PWD/fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf:ro" \
  fluent/fluent-bit:5.1.5 -c /fluent-bit/etc/fluent-bit.conf

Connect Kubernetes and Elasticsearch deliberately

Mount the node log paths required by your runtime, grant the collector narrowly scoped metadata access and choose one owner for each log source. Duplicate tail inputs can duplicate records. With Elasticsearch versions that removed mapping types, check Fluent Bit’s Suppress_Type_Name behavior. Configure trusted TLS, credentials, storage buffering and an appropriate retry policy.

Test the failure path before production

1. Emit known single-line, multiline and malformed application events.
2. Confirm metadata and timestamps in the expected destination.
3. Interrupt the destination, restart the collector and verify buffered recovery.
4. Check duplicates, dropped records, disk limits and retention.

Index separation can help access and retention management but does not inherently make every query faster. See explicit log routing. The smoke test remains a small configuration check. A separate EKS run validated the full collection path, Elasticsearch records and Kibana data-view API.

Sources and further reading

• Fluent Bit multiline parsing
• Fluent Bit Elasticsearch output
• Kubernetes logging architecture

What the live reliability test covered

Fluent Bit 5.1.5 collected the test workload’s CRI logs from EKS nodes and delivered them to Elasticsearch 9.5.5. A controlled service outage exercised filesystem buffering and recovery. The quality fixtures included JSON, a Java exception with two stack frames, and a malformed record. The Java lines became one event and the malformed record remained available.

Collector state and buffers were stored on the disposable nodes for the Pod restart check. The observed JSON sequence had no gaps or duplicates after replacement of the collector Pods. That bounded result does not establish exactly-once delivery under every failure or durability after node loss. An emptyDir is insufficient when the Pod itself is replaced; choose storage that matches the intended failure boundary.