# Root Cause Analysis: 2026-07-15 Kafka file-download consumer-group replay and Nearmap throttling

Incident: `INC-2026-07150002`

## Summary

The rollout changed kafka-file-download from FILE-DOWNLOAD-group to FILE-DOWNLOAD-rmuqop-group. Because the new group had no committed offsets and the consumer subscribes with fromBeginning:true, the rollout replayed retained historical requests. The 66-second overlap of old and new groups amplified processing, exceeded Nearmap rate limits, grew the DLQ, and broke iframe design generation.

## Methods

- Timeline correlation
- Change analysis
- Kafka offset-lineage analysis
- DLQ header aggregation
- Rolling-overlap model

## Root Causes

- **RCA-ROOT-0001** Production consumer-group identity changed across a release, abandoning committed offsets while fromBeginning:true was enabled. Evidence: EV-0003. Actions: ACT-0001, ACT-0005.

## Contributing Factors

- **RCA-CF-0001** All replicas shared INSTANCE_ID=rmuqop, creating one new shared group that ran concurrently with the old group during rollout. Evidence: EV-0003. Actions: ACT-0001.
- **RCA-CF-0002** No request-level idempotency or deduplication prevented duplicate downstream provider calls. Evidence: EV-0003. Actions: ACT-0002.
- **RCA-CF-0003** Retries were process-local and no provider-wide rate limiter or circuit breaker enforced Nearmap capacity across replicas. Evidence: EV-0003, EV-0007. Actions: ACT-0002, ACT-0006.
- **RCA-CF-0004** CD cancellation occurred after the GitOps image commit, creating ambiguity about whether production would still reconcile the change. Evidence: EV-0003. Actions: ACT-0005.
- **RCA-CF-0005** A 30-second per-attempt timeout plus retries can exceed KafkaJS's default 30-second session timeout, while eachMessage does not heartbeat; eviction and rebalance can create duplicate processing. Evidence: EV-0003. Actions: ACT-0010.
- **RCA-CF-0006** Provider responses are fully buffered with arrayBuffer and no size cap. Evidence: EV-0003. Actions: ACT-0010.
- **RCA-CF-0007** Kafka-derived folder/name paths can traverse outside the intended root, and the container runs as root. Evidence: EV-0003. Actions: ACT-0011.
- **RCA-CF-0008** Startup logging of Kafka SASL/SSL objects may expose credentials or private-key material in source or retained logs. Evidence: EV-0003. Actions: ACT-0012.

## Detection Gaps

- **RCA-DG-0001** No deployment gate compared production consumer-group identity and committed offset continuity before rollout. Evidence: EV-0003. Actions: ACT-0005.
- **RCA-DG-0002** The customer UI returned code undefined instead of a traceable correlation ID and actionable failure state. Evidence: EV-0002. Actions: ACT-0008.
- **RCA-DG-0003** Health checks can remain green after run-loop failure because liveness does not await the consumer Promise and the registry check is a one-time settled Promise. Evidence: EV-0003. Actions: ACT-0014.

## Monitoring Gaps

- **RCA-MG-0001** No sufficiently early alert combined group drift, replay pressure, 429 rate, and DLQ growth. Evidence: EV-0003. Actions: ACT-0006.
- **RCA-MG-0002** Deleted-pod stdout was unavailable because the Cloud Logging _Default sink was disabled. Evidence: EV-0003. Actions: ACT-0007.

## Documentation And Release Gaps

- **RCA-DOC-0001** The production offset-identity invariant and safe consumer-group migration procedure were not enforced as a documented release contract. Evidence: EV-0003, EV-0004. Actions: ACT-0001, ACT-0005, ACT-0006.
- **RCA-DOC-0002** Release CI does not make the incident model tests a hard dependency of release, and on-prem overlay patching can report success after string matches become stale. Evidence: EV-0003. Actions: ACT-0013.

## Five Whys

1. Why did design generation fail? File-download requests hit Nearmap throttling and DLQ paths.
2. Why did traffic exceed the provider's capacity? A new Kafka group replayed retained historical requests while the old group still processed current work.
3. Why did the new group replay history? It had no inherited committed offsets and `fromBeginning:true` was enabled.
4. Why did release controls allow that? Consumer-group identity and offset continuity were not hard deployment gates, and cancellation happened after the GitOps commit.
5. Why did replay become a customer outage? There was no request deduplication, shared provider rate budget, header-aware pacing, or fleet-wide circuit breaker.

## Prevention Boundary

Stable group naming protects only valid existing committed offsets. Missing, expired, deleted, or out-of-range offsets can still replay retained history while `fromBeginning:true` remains enabled. #4241 is therefore partial candidate mitigation, not complete prevention.