All case studies
Observability
Modernizing logging with S3-backed Loki and Grafana Alloy
Moving an internal Kubernetes platform off a fragile single-PVC Loki 2.6 setup to an S3-backed Loki 3.x stack with real-time ingestion — no dashboards lost.
- 7 → 30 days
- Retention
- 40 GiB PVC removed
- Single point of failure
- 3-5 s
- Ingestion lag
// the challenge
An internal Kubernetes platform ran Loki 2.6 in SingleBinary mode, with all logs on one 40 GiB PVC — a single point of failure that capped retention at 7 days and made logs unreliable to search when engineers needed them most.
// our approach
- Upgraded to Loki 3.x and moved bulk storage to S3, removing the 40 GiB single-PVC single point of failure.
- Replaced promtail with a Grafana Alloy DaemonSet, cutting ingestion lag to 3-5 seconds behind wall-clock.
- Extended retention from 7 to 30 days using S3 lifecycle policies to control cost.
- Preserved the Grafana datasource UID so every existing dashboard kept working, and hardened the storage IAM user with an explicit Deny pinned to the cluster egress IP.
// the outcome
Engineers now get near-real-time logs (3-5 s behind) with 30 days of searchable history on S3 instead of a fragile 40 GiB volume — and because the datasource UID was preserved, not a single dashboard broke in the cutover.
Have a similar challenge?
Book a 15-minute call and we'll show you where we can help — no pitch, no obligation.