Cheaper logs, longer retention, no single disk to fail

// the challenge

We moved our customer from a full-text-indexed logging stack built on top of opensearch to a label-indexed, S3-backed Grafana Loki and Alloy. Full-text indexing made storage grow very fast and the compute + maintenance overhead was high. On one platform the whole Loki stack sat on a single PVC in SingleBinary mode — a bottleneck and a single point of failure. If that volume filled or failed, logging disappeared exactly when engineers needed it during an incident. Scaling meant paying for more compute and disk, not a cleaner architecture.

// our approach

• Rolled out Grafana Loki with label-only indexing (not full log text), which is the core reason storage and queries stay cheap as volume grows.
• Designed the label scheme for low cardinality so the index stays small and incident queries stay fast.
• Moved bulk storage to S3 and upgraded to a modern Loki release, removing the single-PVC single point of failure and decoupling retention from one disk's size.
• Replaced the old shipper with a Grafana Alloy DaemonSet so ingestion stays a few seconds behind wall-clock — logs are effectively live during an incident.
• Extended retention with S3 lifecycle policies, unified logs and metrics in the existing Grafana (preserving datasource UIDs so dashboards survived the cutover), and hardened storage IAM where the cluster egress IP is pinned.

// the outcome

Logging costs dropped because the system indexes labels instead of full text, and history now lives on S3 instead of a provisioned volume — more retention for less money, with no single disk that can take logging down. During incidents, engineers get near-real-time, filtered queries across environments, and the stack scales with object storage rather than another round of disk and index growth.