← Salesforce Interview Insights

Salesforce·Software Engineer·Onsite - System Design / Architecture·Senior

Senior
Jun 2026

Summary

System design round at Salesforce for a software engineering role, focused entirely on building an async job processing system. Pretty deep dive, they pushed hard on every layer of the design.

Questions Asked (5)

Q1

Design an asynchronous job system where clients can submit long-running tasks, check on their status, cancel them, and retrieve results when done.

System DesignAPI & Integrations
Author's notes

This was the core question and it ate up most of the session.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then design the system in layers: API, job queue, worker pool, and storage. Focus on the core operations (submit, status, cancel, result) and discuss trade-offs around consistency, reliability, and scalability.

Pro tip: Emphasize idempotency and at-least-once processing with deduplication to handle retries gracefully, and discuss how to avoid duplicate job execution—a common pitfall in distributed systems.

1. Clarify Requirements and Scale

Ask about expected throughput, latency requirements, job types, and consistency needs. Define SLAs for job submission, status updates, and result retrieval.

2. Design the API and Data Model

Define RESTful endpoints for submit, status, cancel, and result. Specify the job metadata schema, including status, timestamps, and result location.

3. Architect the Core Components

Outline the job queue (e.g., Kafka, SQS), worker pool for execution, and persistent storage for job state and results. Discuss how components interact and scale.

4. Handle Cancellation and Status Updates

Explain how cancellation requests are propagated (e.g., via a control channel) and how workers check for cancellation. Describe status update mechanisms and consistency guarantees.

5. Address Reliability and Trade-offs

Discuss failure handling, retries, idempotency, and exactly-once vs at-least-once semantics. Compare trade-offs like polling vs webhooks for status updates.

Key Points to Mention

  • Idempotency of job submission and execution to handle retries safely
  • Use of a distributed queue (e.g., Kafka, RabbitMQ) for decoupling and scalability
  • Job state machine (e.g., PENDING, RUNNING, CANCELLED, COMPLETED, FAILED) and transitions
  • Cancellation propagation via a separate control channel or flag in a shared store
  • Result storage options (e.g., object store, database) and retrieval patterns
  • Trade-offs between polling and push notifications (webhooks, SSE) for status updates

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

How would you handle retries for failed jobs, and what happens when a job keeps failing?

System DesignTechnical Trade-offs
Author's notes

Talked through exponential backoff with jitter, which landed well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by outlining a tiered retry strategy with exponential backoff and jitter, then explain how to handle persistent failures through dead-letter queues, alerting, and manual intervention. Emphasize idempotency and observability to ensure reliability and debuggability.

Pro tip: Mention that retries should be idempotent and that you'd use a circuit breaker to prevent cascading failures. Also, highlight the importance of logging context for each retry attempt to aid debugging.

1. Define Retry Policy

Specify the conditions for retrying (e.g., transient errors) and the retry strategy (exponential backoff with jitter, max attempts).

2. Implement Idempotency

Ensure job operations are idempotent so that retries don't cause duplicate side effects.

3. Handle Persistent Failures

After max retries, move the job to a dead-letter queue (DLQ) and trigger alerts for investigation.

4. Monitor and Alert

Set up monitoring on retry rates and DLQ size, with alerts to detect systemic issues.

5. Provide Recovery Path

Define a process for manual intervention, such as replaying DLQ messages after fixing the root cause.

Key Points to Mention

  • Exponential backoff with jitter to avoid thundering herd
  • Idempotency to prevent duplicate processing
  • Dead-letter queue for persistent failures
  • Circuit breaker pattern to prevent cascading failures
  • Monitoring and alerting on retry metrics
  • Manual intervention and replay strategy

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How do you make job submission and execution idempotent so retries don't cause duplicate processing?

System DesignTechnical Trade-offs
Author's notes

Blanked for a second here.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining idempotency in the context of job submission and execution, then explain how to achieve it using idempotency keys, deduplication, and transactional guarantees. Discuss trade-offs between different approaches and how they apply to Salesforce's multi-tenant, distributed environment.

Pro tip: Emphasize that idempotency must be enforced at both submission and execution layers, and mention the importance of idempotent consumers and exactly-once semantics in distributed systems.

1. Define Idempotency and Its Importance

Explain what idempotency means for job processing and why it's critical to prevent duplicate work during retries.

2. Use Idempotency Keys for Submission

Describe how to generate and attach unique idempotency keys to job submissions, and how the system can use them to detect and ignore duplicates.

3. Implement Deduplication and State Tracking

Discuss storing job state and using deduplication mechanisms (e.g., database unique constraints, distributed locks) to ensure each job is processed once.

4. Ensure Idempotent Execution

Explain how to design job execution to be idempotent, such as using upserts, conditional writes, or idempotent operations.

5. Address Trade-offs and Failure Scenarios

Discuss trade-offs between consistency, latency, and complexity, and how to handle partial failures and retries.

Key Points to Mention

  • Idempotency keys and their role in deduplication
  • Transactional guarantees and exactly-once semantics
  • Distributed systems challenges (e.g., network partitions, at-least-once delivery)
  • Database techniques: unique constraints, upserts, conditional updates
  • Idempotent consumers and message deduplication in queues
  • Trade-offs: performance vs. consistency, complexity vs. reliability

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

What storage would you use to track job state, and what are the tradeoffs between a relational database and a NoSQL store here?

System DesignData ModelingTechnical Trade-offs
Author's notes

Went with relational first because transactional state updates matter a lot for correctness in a state machine.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the job state requirements (e.g., volume, consistency needs, query patterns) and then propose a storage solution that balances those needs. Compare relational and NoSQL options across dimensions like consistency, scalability, and query flexibility, and justify your choice with trade-offs. Conclude with a recommendation that aligns with Salesforce's scale and reliability expectations.

Pro tip: Emphasize that the choice depends on the specific job state characteristics—there's no one-size-fits-all. Mention that a hybrid approach (e.g., relational for transactional state, NoSQL for logs/history) is often pragmatic in large systems.

1. Clarify Requirements

Ask about job state volume, update frequency, consistency requirements, and query patterns to ground your answer in concrete needs.

2. Propose a Storage Option

Suggest a primary storage (e.g., relational for ACID, NoSQL for scale) and briefly explain why it fits the requirements.

3. Compare Trade-offs

Discuss key trade-offs: consistency vs. availability, schema flexibility, scalability, query complexity, and operational overhead.

4. Address Salesforce Context

Tie your reasoning to Salesforce's multi-tenant, high-scale environment, mentioning services like DynamoDB, Cassandra, or PostgreSQL.

5. Recommend and Justify

Give a clear recommendation, possibly hybrid, and summarize why it best meets the job state tracking needs.

Key Points to Mention

  • ACID transactions vs. eventual consistency
  • Horizontal scalability and partitioning strategies
  • Schema flexibility and evolution
  • Query patterns and indexing capabilities
  • Operational complexity and maintenance
  • Real-world examples (e.g., DynamoDB for high-scale, PostgreSQL for complex queries)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

Walk me through how workers consume jobs from the queue, and how you'd handle a job that keeps crashing workers.

System DesignTechnical Trade-offs
Author's notes

Covered queue options, worker pools, and visibility timeouts.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining the standard queue consumption model (pull-based, visibility timeout, ack/nack) and then pivot to the failure scenario. For the crashing job, propose a multi-layered defense: poison pill detection, dead-letter queue, and circuit breaker patterns, while emphasizing observability and idempotency.

Pro tip: Mention that you'd add a 'crash loop' alert with a threshold (e.g., 3 crashes in 5 minutes) and automatically quarantine the job to a DLQ for manual inspection, preventing queue backup. Also, highlight that you'd review the job's payload and logs to distinguish between a bad message and a systemic bug.

1. Describe the happy path

Explain how workers pull jobs from the queue (e.g., long polling, visibility timeout), process them, and acknowledge success or failure. Mention at-least-once delivery and idempotency.

2. Identify the crash scenario

Define what 'keeps crashing workers' means: the job causes an unhandled exception, OOM, or infinite loop, leading to repeated redelivery and worker restarts.

3. Implement immediate safeguards

Propose a retry limit with exponential backoff and a dead-letter queue (DLQ) after N attempts. Use a circuit breaker to stop processing that job type if failures exceed a threshold.

4. Add observability and alerting

Instrument the worker to log job ID, payload, and stack trace on crash. Set up alerts for repeated failures and monitor DLQ depth.

5. Root cause and remediation

Analyze the DLQ job to fix the underlying bug (e.g., malformed input, dependency failure). Consider adding validation or sandboxing for untrusted payloads.

Key Points to Mention

  • Visibility timeout and ack/nack semantics in queue systems (e.g., SQS, RabbitMQ).
  • Idempotency keys to handle duplicate processing due to retries.
  • Dead-letter queue (DLQ) with a max receive count to isolate poison pills.
  • Exponential backoff with jitter to avoid thundering herd on retries.
  • Circuit breaker pattern to prevent cascading failures.
  • Observability: structured logging, metrics (crash rate, DLQ size), and alerting.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.