Automate Elasticsearch Index Creation with Scoped Templates

Elasticsearch can create an index when an authorized client writes its first document, subject to automatic-creation settings. Configure a matching template first so that mappings and shard settings are intentional. Limit index-name inputs: arbitrary application labels can create excessive shards or invalid names.

Verify the core behavior locally

Start the disposable container and apply the template from our Elasticsearch operations lab. The following creates a second index through a document write.

curl -fsS -X PUT "$ES_URL/cloudhero-lab-auto/_doc/1?refresh=true" \
  -H 'Content-Type: application/json' -d '{"message":"automatic creation"}'
curl -fsS "$ES_URL/cloudhero-lab-auto/_settings"
curl -fsS "$ES_URL/cloudhero-lab-auto/_mapping"

Check that the template’s settings and mappings were applied. Production permission policies may intentionally prohibit automatic index creation.

Use current Filebeat input configuration

The old container input is based on the deprecated log input and is disabled by default in Filebeat 9. Use filestream with a container parser. This is an integration fragment, not a complete DaemonSet: it requires node log mounts, an appropriate output and permissions.

filebeat.inputs:
  - type: filestream
    id: kubernetes-container-logs
    prospector.scanner.symlinks: true
    paths:
      - /var/log/containers/*.log
    parsers:
      - container:
          stream: all
          format: auto

Keep routing bounded

Allow only known application and environment values, sanitize the names and send unmatched events to a defined fallback. Preserve the original message on parse failure. Evaluate a data stream or a small number of indices against retention and access requirements before creating one index per application per day.

Deploy and verify the whole pipeline

The original Elastic Helm repository is archived. Select supported Elastic deployment tooling, such as ECK where appropriate, or your managed service’s instructions. Validate Filebeat input, Kubernetes metadata enrichment, Logstash parsing, TLS and credentials together. Lifecycle management availability depends on the deployment; it is not universally a paid-only feature.

1. Send one known event and find it in the expected index.
2. Send an unknown application label and malformed event; verify fallback handling.
3. Restart the collector and check for loss or duplicates.
4. Verify lifecycle and retention on the intended data only.

The API lab checks automatic creation separately. The live EKS run also collected CRI-format container logs with Filebeat 9.5.5 filestream, decoded NDJSON fields, enriched Kubernetes namespace metadata and delivered through Logstash 9.5.5 into the expected Elasticsearch index. Delivery recovered after a controlled destination service outage. A Kibana data view was created against the test indices. Remove the local container when finished.

Sources and further reading

• Elasticsearch index templates
• Filebeat container input migration
• Archived Elastic Helm charts
• ILM availability

Match metadata to the CRI log path

The default log-path matcher did not enrich the test records. Set the matcher to the path actually collected; the service account also needs get/list/watch access to the Pods being enriched.

processors:
  - add_kubernetes_metadata:
      host: ${NODE_NAME}
      namespace: logging-live
      scope: node
      default_matchers.enabled: false
      matchers:
        - logs_path:
            logs_path: /var/log/containers/
      add_resource_metadata:
        node: {enabled: false}
        namespace: {enabled: false}

Here logging-live is the test namespace. Set it to the namespace you intend to collect and align the Role/RoleBinding. The DaemonSet supplied NODE_NAME from spec.nodeName. See Elastic’s metadata processor reference.

Keep the collector state with the collector lifecycle

The live restart check stored Filebeat’s registry on its disposable node so the state survived collector Pod replacement. The directory also had to be writable by the collector’s UID; the test initialized ownership to UID/GID 1000. Verify the image’s actual user and provision only its dedicated state directory. An emptyDir alone does not survive replacement of the Pod. Choose a state and buffering design for the failures you need to tolerate, and test duplicates and gaps. A successful Pod restart test does not validate recovery after losing the node or its disk.