# Technical Incident Report: 2026-07-15 Kafka file-download consumer-group replay and Nearmap throttling

Incident ID: `INC-2026-07150002`  
Severity: `SEV2` / customer `major`  
Category: `Partial Outage`  
Environment: `Aerialytic production main`  
Status: **Monitoring / Remediation Pending**  
Incident Commander: Aerialytic Incident Response  
Started: `2026-07-15T19:38:32.823Z`  
Detected: `2026-07-15T19:46:51.403Z`  
Private Slack record: [#incident-reports](https://aerialytic.slack.com/archives/C0BF16AKVQX/p1784151610481449)

## Executive Technical Summary

A production rollout changed kafka-file-download from the established FILE-DOWNLOAD-group consumer identity to FILE-DOWNLOAD-rmuqop-group while retaining fromBeginning:true. The new group had no committed offsets and replayed retained historical download requests during a 66-second old/new rolling overlap. Duplicate traffic caused Nearmap HTTP 429 throttling, file-download DLQ growth, and customer-facing iframe design-generation failures. Production was rolled back to version 2.0.0 and is currently healthy, but permanent remediation, exact impact enumeration, and recovery of failed jobs remain pending.

Production rollback and permanent remediation are separate claims. Version `2.0.0` is currently healthy, while candidate PR [#4241](https://github.com/Aerialytic/Monorepo/pull/4241) is not live and is only a partial mitigation.

## Confirmed Impact

| Category | Severity | Confidence | Assessment |
| --- | --- | --- | --- |
| availability | high | confirmed | A core iframe design-generation workflow was unavailable or unreliable for multiple customers while other production surfaces continued operating. |
| customerExperience | critical | high | Customers attempting to generate designs in embedded iframe workflows saw an unhelpful 'Aerialytic error (code undefined)' message. Multiple-customer impact is confirmed; exact deduplicated impact remains pending. |
| operational | high | confirmed | The new Kafka group replayed retained requests, amplified provider traffic, produced 2,222 DLQ records during the bad-release window, and requires deduplicated recovery work. |
| integrity | low | medium | No source-data corruption is currently evidenced. Duplicate at-least-once processing outcomes occurred, and exact request-level deduplication remains unproven. |
| confidentiality | low | medium | No evidence of unauthorized access, compromise, customer-data exposure, or exfiltration was observed. A separate audit found startup logging of Kafka SASL/SSL objects that may include credential or private-key material; exposure assessment and scoped rotation are urgent and pending. |

During `2.0.2` from `2026-07-15T19:38:32.823Z` through the final bad pod at `2026-07-15T20:01:56Z`, the file-download DLQ recorded exactly `2222` outcomes: `1267` HTTP 429 and `955` HTTP 400. Before rollback began, `80` new source messages produced `4576` outcomes. Approximately `4495` were historical replay; one outcome remains unattributed.

These are outcome counts, not unique customer or job counts. Exact deduplicated impact remains pending.

## Root Cause

The rollout changed kafka-file-download from FILE-DOWNLOAD-group to FILE-DOWNLOAD-rmuqop-group. Because the new group had no committed offsets and the consumer subscribes with fromBeginning:true, the rollout replayed retained historical requests. The 66-second overlap of old and new groups amplified processing, exceeded Nearmap rate limits, grew the DLQ, and broke iframe design generation.

## Detailed Findings

## Root Causes

- **RCA-ROOT-0001** Production consumer-group identity changed across a release, abandoning committed offsets while fromBeginning:true was enabled. Evidence: EV-0003. Actions: ACT-0001, ACT-0005.

## Contributing Factors

- **RCA-CF-0001** All replicas shared INSTANCE_ID=rmuqop, creating one new shared group that ran concurrently with the old group during rollout. Evidence: EV-0003. Actions: ACT-0001.
- **RCA-CF-0002** No request-level idempotency or deduplication prevented duplicate downstream provider calls. Evidence: EV-0003. Actions: ACT-0002.
- **RCA-CF-0003** Retries were process-local and no provider-wide rate limiter or circuit breaker enforced Nearmap capacity across replicas. Evidence: EV-0003, EV-0007. Actions: ACT-0002, ACT-0006.
- **RCA-CF-0004** CD cancellation occurred after the GitOps image commit, creating ambiguity about whether production would still reconcile the change. Evidence: EV-0003. Actions: ACT-0005.
- **RCA-CF-0005** A 30-second per-attempt timeout plus retries can exceed KafkaJS's default 30-second session timeout, while eachMessage does not heartbeat; eviction and rebalance can create duplicate processing. Evidence: EV-0003. Actions: ACT-0010.
- **RCA-CF-0006** Provider responses are fully buffered with arrayBuffer and no size cap. Evidence: EV-0003. Actions: ACT-0010.
- **RCA-CF-0007** Kafka-derived folder/name paths can traverse outside the intended root, and the container runs as root. Evidence: EV-0003. Actions: ACT-0011.
- **RCA-CF-0008** Startup logging of Kafka SASL/SSL objects may expose credentials or private-key material in source or retained logs. Evidence: EV-0003. Actions: ACT-0012.

## Detection Gaps

- **RCA-DG-0001** No deployment gate compared production consumer-group identity and committed offset continuity before rollout. Evidence: EV-0003. Actions: ACT-0005.
- **RCA-DG-0002** The customer UI returned code undefined instead of a traceable correlation ID and actionable failure state. Evidence: EV-0002. Actions: ACT-0008.
- **RCA-DG-0003** Health checks can remain green after run-loop failure because liveness does not await the consumer Promise and the registry check is a one-time settled Promise. Evidence: EV-0003. Actions: ACT-0014.

## Monitoring Gaps

- **RCA-MG-0001** No sufficiently early alert combined group drift, replay pressure, 429 rate, and DLQ growth. Evidence: EV-0003. Actions: ACT-0006.
- **RCA-MG-0002** Deleted-pod stdout was unavailable because the Cloud Logging _Default sink was disabled. Evidence: EV-0003. Actions: ACT-0007.

## Documentation And Release Gaps

- **RCA-DOC-0001** The production offset-identity invariant and safe consumer-group migration procedure were not enforced as a documented release contract. Evidence: EV-0003, EV-0004. Actions: ACT-0001, ACT-0005, ACT-0006.
- **RCA-DOC-0002** Release CI does not make the incident model tests a hard dependency of release, and on-prem overlay patching can report success after string matches become stale. Evidence: EV-0003. Actions: ACT-0013.

## Containment And Current State

- Production is rolled back to `ghcr.io/aerialytic/kafka-file-download:2.0.0`.
- `main/kafka-file-download-rmuqop-deployment` was last observed `3/3 Ready` with zero restarts.
- The latest 15-minute sample had 3 completed downloads, 0 fresh 429, 0 DLQ sends, and 0 processing errors.
- Provider throttling continued after rollback until the last observed 429 at `2026-07-15T20:36:09.138Z`.
- Failed-job recovery and exact customer/job inventory remain pending.

## Remediation Readiness

Current #4241 head: `dbd2126e92f70044696312e43200b3082b7eafd1`.

What the candidate does:

- Preserves the established production group when valid committed offsets exist.
- Retries only Effect HTTP response errors with status 429 or 5xx.
- Uses jittered exponential backoff with a one-second base and 3 retries / 4 total attempts.
- Does not retry other 4xx, decode failures, or unrelated errors.
- Adds environment validation, signed-URL query redaction, locality tests, and offset-contract documentation.

Release blockers and adjacent remediation:

- Provider-wide distributed rate gate
- X-RateLimit header-aware pacing and reset delay
- Provider circuit breaker
- Fleet concurrency budget
- Consumer heartbeat and ownership-safe slow-request handling
- Bounded response buffering
- Offset continuity preflight for missing, expired, deleted, or out-of-range commits
- Request-level deduplication or idempotency and crash-consistent output/offset handling
- Path traversal removal and non-root runtime
- Kafka SASL/SSL credential-log removal, exposure assessment, and scoped rotation
- Run-loop-aware health checks
- Actual-artifact CI/CD enforcement
- Next/Staging overlay and storage-locality verification

Full CI run [29450818576](https://github.com/Aerialytic/Monorepo/actions/runs/29450818576) was queued at publication time. It is not documented as green.

## Vendor Contract

- Nearmap recommends exponential backoff for HTTP 429 and 5xx, not other 4xx: [failed API request guidance](https://help.nearmap.com/kb/articles/1297-capturing-information-for-failed-api-requests).
- Nearmap documents endpoint/account-specific `X-RateLimit-Limit`, `X-RateLimit-Remaining`, and `X-RateLimit-Reset` response headers: [API standard](https://developer.nearmap.com/docs/nearmap-api-standards).

## Evidence Boundary

Kafka inspection was read-only and parsed only x-topic/x-error headers. Payloads and signed URLs were not displayed or persisted.

Unproven:

- Exact per-pod record attribution
- Incidental at-least-once duplicates
- Unique business request IDs
- Deduplicated customer and job impact
- Automatic recovery of failed jobs
- Live provider behavior under PR #4241

## Action Register

| ID | Priority | Status | Owner | Due | Action | Verification |
| --- | --- | --- | --- | --- | --- | --- |
| ACT-0001 | P0 | in_progress | Kafka File Download Service Owners | 2026-07-16 | Complete, merge, release, and verify the #4241 remediation set | Required adjacent blockers are resolved or linked to explicit approved follow-ups; PR merged; exact commit built and promoted; production group remains FILE-DOWNLOAD-group; valid offset continuity is proven; rollout has no replay surge, 429 spike, DLQ growth, credential/signed-URL leakage, or stale on-prem overlay; CI and client-like checks pass. |
| ACT-0002 | P0 | accepted | Aerialytic Data Recovery and Kafka Platform | 2026-07-16 | Recover failed jobs with deduplication and provider-wide rate bounds | Signed recovery manifest; deduplicated request count; provider-rate budget; dry-run approval; bounded execution receipts; customer workflow verification; residual DLQ disposition. |
| ACT-0003 | P0 | in_progress | Aerialytic Incident Response and Data Engineering | 2026-07-16 | Produce exact impacted customer and job inventory | Versioned deduplicated inventory with query hashes, inclusion/exclusion rules, unresolved records, and customer/job counts. |
| ACT-0004 | P0 | accepted | Aerialytic Customer Success and Support | 2026-07-16 | Complete customer recovery and approved communications | Approved communication log and per-customer recovery confirmation linked to the deduplicated impact inventory. |
| ACT-0005 | P1 | accepted | Aerialytic Release Engineering | 2026-07-17 | Add offset-continuity and cancellation deployment gates | Negative CI fixtures for group drift; approved migration override contract; cancellation/reconcile integration test; production release evidence includes before/after group and offset checks. |
| ACT-0006 | P1 | in_progress | Aerialytic SRE and Observability | 2026-07-17 | Ship 429, DLQ, replay-pressure, and group-drift alerts with runbook #4244 | Issue #4244 closed with alert-rule tests, synthetic threshold exercises, dashboards, paging destinations, and runbook drill evidence. |
| ACT-0007 | P1 | accepted | Aerialytic Cloud Platform and Security | 2026-07-22 | Restore production workload log retention | Retention policy approved; deleted-pod log retrieval test passes; signed URL and credential redaction tests pass; access and cost controls documented. |
| ACT-0008 | P1 | accepted | Aerialytic Iframe Product and Frontend | 2026-07-22 | Add customer-facing error correlation and actionable design failure states | Browser test reproduces a controlled provider failure and verifies a stable safe code, correlation ID, support path, and matching server trace. |
| ACT-0009 | P0 | accepted | Kafka File Download Service Owners and SRE | 2026-07-17 | Add provider-wide header-aware rate control and circuit breaking | Multi-replica load tests prove one shared budget, header-aware pause/reset behavior, bounded attempts, circuit-open behavior, and no provider overrun at 1/3/5-pod scale. |
| ACT-0010 | P0 | accepted | Kafka Platform and File Download Service Owners | 2026-07-17 | Harden Kafka consumer ownership, buffering, and crash consistency | Tests cover responses over the size cap, 30-second-plus provider stalls, rebalances during eachMessage, crash points around output/DLQ and offset commit, and duplicate request identifiers. |
| ACT-0011 | P0 | accepted | Kafka File Download Service Owners and Application Security | 2026-07-16 | Remove file path traversal and root-container exposure | Traversal, encoded traversal, absolute-path, symlink-escape, and valid nested-path tests pass; runtime securityContext proves non-root and least-privilege mounts. |
| ACT-0012 | P0 | in_progress | Aerialytic Security and Cloud Platform | 2026-07-16 | Remove credential-bearing startup logs and rotate exposed Kafka/TLS material | Code/log scans show no secret-bearing object logs; exposure assessment is signed; affected credentials are scoped and rotated; old credentials fail; workloads recover without printing secret values. |
| ACT-0013 | P1 | accepted | Aerialytic Release Engineering | 2026-07-17 | Make incident contracts hard release gates on the actual artifact | A deliberately broken group, retry, secret-log, health, and stale-overlay fixture each blocks release before deployment; promoted image digest and runtime overlay hashes match the tested release evidence. |
| ACT-0014 | P1 | accepted | Kafka File Download Service Owners | 2026-07-18 | Make health checks reflect the live consumer run loop | Fault-injection tests terminate or stall the consumer and prove readiness/liveness transitions, alerting, and bounded restart behavior. |