← Baseten Interview Insights

Baseten·Software Engineer·Onsite - System Design / Architecture·Senior

Senior
May 2026

Summary

Follow-up system design question at Baseten, focused on correctness verification for a parallel S3 chunk downloader. Pretty deep dive into checksums and testing strategy, felt more like a design review than a typical interview.

Questions Asked (3)

Q1

After building a parallel S3 chunk downloader, how do you verify that every chunk arrived correctly and that the reassembled file actually matches what the server has?

System DesignTechnical Trade-offsAPI & Integrations
Author's notes

This is where I started rambling a bit.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining per-chunk integrity checks (e.g., checksums) to catch corruption during transfer, then describe whole-file verification (e.g., comparing ETag or computing a full-file hash) to ensure the reassembled file matches the server's version. Emphasize handling edge cases like retries, partial failures, and multipart upload semantics.

Pro tip: Mention that S3's ETag for multipart uploads is not a simple MD5 of the whole file, so you need to either compute the multipart ETag yourself or use a separate checksum like SHA-256 stored in metadata. This shows deep AWS knowledge and avoids a common pitfall.

1. Verify each chunk during download

Use per-chunk checksums (e.g., MD5, CRC32, or SHA-256) provided by S3 or stored in metadata to validate each chunk immediately after download. Retry failed chunks automatically.

2. Reassemble and verify file integrity

After all chunks are downloaded, reassemble the file and compute a whole-file checksum. Compare it against the expected checksum from the server (e.g., S3 ETag for single-part uploads or a custom checksum for multipart).

3. Handle multipart ETag nuances

If the file was uploaded via multipart upload, the ETag is not a simple MD5. Either compute the multipart ETag by concatenating chunk MD5s and hashing, or rely on a separate checksum stored in object metadata.

4. Implement retry and error handling

Design the downloader to retry failed chunks with exponential backoff, and log discrepancies. If verification fails after retries, fail gracefully and report which chunk(s) are corrupt.

5. Test with fault injection

Simulate network errors, corrupted chunks, and partial downloads to ensure your verification logic catches issues. Use tools like Toxiproxy or mock S3 responses.

Key Points to Mention

  • Per-chunk checksums (MD5, CRC32, SHA-256) for early corruption detection
  • Whole-file checksum comparison (e.g., SHA-256) against server-provided value
  • S3 ETag semantics: single-part vs multipart uploads
  • Retry mechanisms with exponential backoff for failed chunks
  • Concurrency control and ordering to avoid race conditions during reassembly
  • Logging and monitoring for verification failures

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

If a chunk checksum or the final file size doesn't match what you expected, do you retry or fail fast, and what drives that decision?

System DesignTechnical Trade-offsRoot Cause Analysis
Author's notes

Went with retry with exponential backoff for transient network issues, fail fast if the server-side checksum itself is wrong because that's a data integrity problem not a flakiness problem.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around a decision framework that balances correctness, cost, and user impact. Explain that the choice depends on factors like idempotency, retry cost, and whether the mismatch indicates a transient or permanent failure. Emphasize that you would instrument and log mismatches to inform the decision and avoid blind retries.

Pro tip: Mention that you'd implement a bounded retry with exponential backoff and jitter, but only if the operation is idempotent and the error is likely transient. Also, highlight that you'd add a circuit breaker to prevent cascading failures if mismatches persist.

1. Classify the error

Determine if the mismatch is likely transient (e.g., network glitch, partial write) or permanent (e.g., data corruption, bug). Check error codes, logs, and patterns.

2. Assess retry safety

Verify if the operation is idempotent and if retrying could cause side effects or duplicate work. Consider the cost of retry in terms of latency, resources, and user experience.

3. Define retry policy

If retrying, set a bounded number of attempts with exponential backoff and jitter. If not, fail fast and surface the error with enough context for debugging.

4. Monitor and adapt

Log all mismatches and retry outcomes. Use metrics to detect patterns and adjust thresholds or policies dynamically.

5. Communicate and recover

If failing fast, ensure the system degrades gracefully and alerts are triggered. If retrying, provide feedback to the user or upstream system about the delay.

Key Points to Mention

  • Idempotency of the operation and whether retries are safe
  • Cost of retry vs. cost of failure (latency, resources, user impact)
  • Transient vs. permanent error classification
  • Bounded retries with exponential backoff and jitter
  • Circuit breakers and fallback mechanisms
  • Observability: logging, metrics, and alerting for mismatches

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How would you unit and integration test this downloader, specifically around chunk failures and reassembly correctness?

System DesignTechnical Trade-offs
Author's notes

Deterministic fakes for the S3 client, inject faults on specific chunk indices to simulate corruption or timeout.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the downloader's architecture and failure modes, then outline a layered testing strategy: unit tests for chunk-level logic (e.g., retries, checksums) and integration tests for end-to-end reassembly under simulated failures. Emphasize deterministic, fast tests using mocks and fault injection to cover edge cases like partial writes and out-of-order chunks.

Pro tip: Use property-based testing to generate random chunk sequences and verify reassembly invariants, and inject faults at the network layer (e.g., via a proxy) to simulate real-world chunk failures without modifying production code.

1. Clarify Requirements and Architecture

Ask about the downloader's design: chunk size, retry policy, storage, and reassembly method. This ensures your testing strategy targets the actual failure points.

2. Unit Test Chunk-Level Logic

Test individual components like chunk fetching, checksum validation, and retry logic in isolation using mocks for network and storage. Cover edge cases: timeouts, corrupted data, and max retries.

3. Integration Test Reassembly Correctness

Simulate multi-chunk downloads with controlled failures (e.g., drop a chunk, delay a chunk) and verify the final file matches the original. Use a test server or fault-injection proxy to orchestrate scenarios.

4. Test Failure Recovery and Concurrency

Verify that the downloader resumes correctly after failures, handles out-of-order chunk arrival, and avoids race conditions when writing chunks concurrently.

5. Automate and Measure Coverage

Integrate tests into CI, use code coverage to identify untested paths, and add performance tests for large files to ensure reassembly scales.

Key Points to Mention

  • Mocking network and storage layers to isolate unit tests
  • Fault injection techniques (e.g., dropping chunks, corrupting data)
  • Checksum or hash verification for reassembly correctness
  • Testing retry logic and exponential backoff
  • Concurrency issues: out-of-order chunks, partial writes, and race conditions
  • Property-based testing to validate reassembly invariants

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.