Elasticsearch Operations: Templates, Reindexing and Recovery

Posted on April 16, 2020

Updated

Manage Elasticsearch by checking cluster health, disk allocation, mappings, lifecycle policies and recovery before changing data. The examples below use a disposable Elasticsearch 9.5.5 container and explicitly named lab indices. They replace the original broad deletion loops.

1. Diagnose disk pressure before clearing blocks

The usual disk thresholds are a low watermark at 85%, high at 90%, and flood stage at 95%, subject to configured headroom. Flood stage can apply a read-only-allow-delete block; the low watermark alone does not stop all ingestion. Fix the underlying capacity or retention issue and inspect effective settings. A 403 can have other causes, including authorization.

2. Create a scoped composable template

For this loopback-only local lab, security is disabled inside the container. Do not expose that configuration on a network or use it for production. Production clients need authentication and trusted TLS. Allow memory for Docker and the other services on your machine.

docker run -d --name cloudhero-es-lab --memory=1536m \
  -p 127.0.0.1:19200:9200 \
  -e discovery.type=single-node -e xpack.security.enabled=false \
  -e ES_JAVA_OPTS='-Xms512m -Xmx512m' \
  docker.elastic.co/elasticsearch/elasticsearch:9.5.5
export ES_URL=http://127.0.0.1:19200
# Wait up to two minutes, then fail visibly if startup did not succeed.
for ((attempt=1; attempt<=60; attempt++)); do
  if curl -fsS "$ES_URL/_cluster/health?wait_for_status=yellow&timeout=1s"; then
    break
  fi
  sleep 2
done
curl -fsS "$ES_URL/_cluster/health?wait_for_status=yellow&timeout=5s"
curl -fsS -X PUT "$ES_URL/_index_template/cloudhero-lab" \
  -H 'Content-Type: application/json' -d '{
  "index_patterns": ["cloudhero-lab-*"],
  "priority": 501,
  "template": {
    "settings": {"number_of_shards": 1, "number_of_replicas": 0},
    "mappings": {"properties": {
      "@timestamp": {"type": "date"},
      "message": {"type": "text"},
      "location": {"type": "geo_point"}
    }}
  }
}'
curl -fsS -X PUT "$ES_URL/cloudhero-lab-source/_doc/1?refresh=true" \
  -H 'Content-Type: application/json' -d '{
  "@timestamp": "2026-10-30T00:00:00Z",
  "message": "demo", "location": {"lat": 44.43, "lon": 26.10}
}'
curl -fsS "$ES_URL/cloudhero-lab-source/_mapping"

The zero-replica setting is for this single-node lab. Production replica counts follow availability requirements. A new template affects future indices; dynamic settings such as replica count can often be changed on existing indices without reindexing.

3. Reindex into an explicit destination

Reindex copies documents; it does not copy every source setting or mapping. Create or template the destination first. Source can include more than one index. Quiesce writes or define a synchronization strategy before a production cutover.

curl -fsS -X PUT "$ES_URL/cloudhero-lab-copy"
curl -fsS -X POST "$ES_URL/_reindex?refresh=true" \
  -H 'Content-Type: application/json' -d '{
  "source": {"index": "cloudhero-lab-source"},
  "dest": {"index": "cloudhero-lab-copy", "op_type": "create"}
}'
curl -fsS "$ES_URL/cloudhero-lab-source/_count"
curl -fsS "$ES_URL/cloudhero-lab-copy/_count"

Inspect reindex failures, counts, representative searches and mappings. Counts alone are insufficient for a live changing dataset. Retain the source until a separate approved cutover and rollback review.

4. Exercise snapshot restore separately

Verify repository access and version compatibility. Restore only intended indices and avoid restoring global state into an existing cluster without a deliberate plan. In the live EKS test, Elasticsearch 9.5.5 wrote a snapshot to an encrypted private S3 bucket using workload IAM. A separate Elasticsearch cluster restored it through a read-only IAM role and read-only repository. Document count and the geo_point mapping were preserved; the source remained intact.

5. Configure GeoIP and Kibana explicitly

The mapped location field above is a geo_point. Match its path to your ingest pipeline; ECS-based GeoIP fields may use another path. In Kibana, create a data view using its current data-view API and the intended space, permissions and timestamp field. Kibana and GeoIP enrichment are separate integration checks.

Remove only the disposable lab

docker rm -f -v cloudhero-es-lab

Use lifecycle management appropriate to your deployment rather than a periodic script deleting all indices. See automatic index creation for a scoped ingestion pattern.

Sources and further reading

Check workload identity and the destination repository

The live test used short-lived EKS Pod Identity credentials, with no static AWS keys. Elasticsearch’s restricted file access required the projected authorization token inside its configuration directory. When overriding the injected token path, the credential endpoint also had to be supplied explicitly. Validate this behavior against your Elasticsearch and SDK versions rather than assuming an injected environment variable is sufficient.

A second cluster should register the shared repository as read-only. This prevents it from modifying repository metadata owned by the writer. In our test its IAM role also had only list/read access to the test repository prefix.

PUT /_snapshot/readonly-lab
{
  "type": "s3",
  "settings": {
    "bucket": "your-approved-snapshot-bucket",
    "base_path": "es-snapshots",
    "region": "eu-central-1",
    "readonly": true
  }
}

The destination still needs a compatible Elasticsearch version, appropriate index privileges and access to any required encryption keys. Registering a repository does not alone establish that restore works: check the restored records, mappings and application queries. The live test also created and read a Kibana 9.5.5 data view through its current API.