← Illumio Interview Insights

Illumio·Software Engineer·Onsite - System Design / Architecture·Senior

Senior
May 2026

Summary

System design round at Illumio focused almost entirely on Kafka internals. One scenario-based question that went pretty deep into failure recovery and offset management. Felt more like a distributed systems seminar than a typical interview.

Questions Asked (1)

Q1

A Kafka consumer is processing messages from a partition. Offset 2 fails due to a handler exception, but offset 4 is successfully processed and committed. How do you recover and reprocess offset 2 without losing data or re-processing what already succeeded?

System DesignTechnical Trade-offsRoot Cause Analysis
Author's notes

This one is nastier than it looks.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, clarify that the scenario implies auto-commit is enabled, which caused offset 4 to be committed after offset 2 failed. Then, propose a recovery strategy that involves resetting the consumer group offset to 2, but with safeguards to avoid reprocessing offset 4, such as using a dead-letter queue or idempotent processing.

Pro tip: Mention that in production, you should disable auto-commit and manually commit offsets only after successful processing to prevent this issue. Also, consider using a dead-letter topic for failed messages to isolate and reprocess them without affecting the main consumer group.

1. Diagnose the offset commit behavior

Explain that with auto-commit, offsets are committed periodically based on the latest processed message, so offset 4 being committed means offset 2 is skipped. Confirm that the consumer group's committed offset is now 5 (next to read).

2. Stop the consumer and assess impact

Halt the consumer to prevent further commits. Determine if offset 4's processing is idempotent or if reprocessing it would cause issues. If not idempotent, you need a way to skip it during reprocessing.

3. Reset offset to reprocess from offset 2

Use kafka-consumer-groups.sh to reset the consumer group offset to 2. Alternatively, seek to offset 2 programmatically. Ensure the consumer is configured to handle the reprocessing of offset 4 gracefully.

4. Implement idempotent processing or filtering

Make the message handler idempotent so reprocessing offset 4 doesn't cause duplicate side effects. Or, maintain a separate store of processed offsets and skip offset 4 if already processed.

5. Prevent recurrence with manual commits and DLQ

Switch to manual offset commits after successful processing. Introduce a dead-letter queue for failed messages to isolate them and allow reprocessing without disrupting the main flow.

Key Points to Mention

  • Auto-commit vs manual commit and their implications
  • Consumer group offset management and resetting offsets
  • Idempotent message processing to handle duplicates
  • Dead-letter queue (DLQ) for failed messages
  • Kafka consumer seek() and commitSync() APIs
  • Monitoring and alerting for consumer lag and failures

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.