← Atlassian Interview InsightsSoftware Engineer
I got the basic shape of it pretty quickly: POST to create a job, GET to poll status, return discovered images once done.
Start by defining the REST API contract clearly — endpoints for batch submission, status polling, and result retrieval — then walk through the async processing architecture that handles crawling at scale. Ground your design in practical trade-offs like rate limiting, depth control, and idempotency, which signal real-world engineering maturity.
Pro tip: Atlassian values systems that are resilient and observable; proactively mention how you'd handle crawl failures, retries with exponential backoff, and how operators would monitor job health via metrics or webhooks — this shows you think beyond the happy path.
Specify the core endpoints: POST /jobs to submit a batch of URLs (returning a job ID), GET /jobs/{jobId} for status polling, and GET /jobs/{jobId}/results for paginated image results. Clarify request/response schemas including fields like status (PENDING, IN_PROGRESS, COMPLETED, FAILED), submitted URLs, and extracted image URLs.
Explain why crawling must be asynchronous — network I/O and unknown link depth make synchronous responses impractical. Describe a job queue (e.g., SQS, Kafka, or RabbitMQ) where each URL becomes a task consumed by a pool of crawler workers.
Define the data model: a Jobs table (jobId, status, createdAt, totalUrls, completedUrls) and an Images table (jobId, sourceUrl, imageUrl, discoveredAt). Discuss using a relational DB for job metadata and optionally a key-value store or blob storage for large result sets.
Address crawl depth limits, visited URL deduplication (using a distributed set like Redis), robots.txt compliance, and per-domain rate limiting to avoid being blocked or causing harm. Mention timeouts and retry logic with exponential backoff for failed URLs.
Discuss horizontal scaling of crawler workers, idempotent job submission to prevent duplicate processing, and optional webhook callbacks so clients aren't forced to poll. Highlight metrics to expose (queue depth, crawl latency, error rates) and how partial results can be streamed before a job fully completes.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Structure your answer by walking through each major component in logical order: the URL frontier/queue, the worker pool, and deduplication, explaining the design decisions and trade-offs at each layer. Ground your architecture in concrete data structures and distributed systems primitives rather than staying abstract. Tie your choices back to real-world constraints like scale, fault tolerance, and politeness policies.
Pro tip: Demonstrate senior-level thinking by proactively addressing the 'politeness' problem — rate-limiting crawls per domain to avoid hammering servers — as this is a nuance many candidates miss and signals you've thought beyond the happy path.
Briefly establish the scale you're designing for (e.g., billions of URLs, distributed across many machines) to justify your architectural choices. Clarify whether this is a focused crawler or a broad web crawler, as this shapes every subsequent decision.
Describe a multi-priority, multi-queue frontier that separates URLs by domain to enforce politeness and prioritize high-value pages. Explain using a combination of a priority queue (for importance scoring) backed by persistent storage like Kafka or Redis to survive crashes and handle backpressure.
Explain a pool of async worker threads or processes that pull URLs from the frontier, fetch content, parse links, and emit new URLs back to the queue. Discuss using async I/O (e.g., Python asyncio or Java NIO) to maximize throughput since crawling is heavily I/O-bound, not CPU-bound.
Walk through a layered deduplication strategy: a Bloom filter for fast, memory-efficient in-memory checks as the first gate, backed by a distributed key-value store (e.g., Redis or Cassandra) storing URL hashes for authoritative deduplication. Acknowledge the Bloom filter's false-positive trade-off and explain why it's acceptable here.
Explain how workers should be stateless so failed jobs can be retried, and how the frontier should use acknowledgment-based dequeuing to prevent URL loss. Describe per-domain rate limiting using token buckets or delay queues to respect robots.txt and avoid IP bans.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked for a second on robots.txt parsing and just said I'd fetch and cache it at the start of crawling each domain.
Frame your answer around the core principles of being a 'good citizen' on the web, covering both technical mechanisms (rate limiting, robots.txt parsing) and architectural decisions (distributed coordination, backoff strategies). Structure your response by first defining the problem constraints, then walking through the implementation layers from protocol compliance to infrastructure design. Demonstrate awareness of trade-offs between crawl speed and server impact.
Pro tip: Mention that robots.txt should be cached with a TTL (e.g., 24 hours) and fetched before any other page on a domain — this signals you understand real-world operational concerns, not just the happy path. Bonus points if you bring up the Crawl-Delay directive in robots.txt and how it should override your default rate limits.
Explain how to fetch and parse robots.txt for each domain before crawling any URLs, respecting Disallow rules, Allow overrides, and the Crawl-Delay directive. Discuss caching the parsed result with an appropriate TTL to avoid redundant fetches.
Describe maintaining a per-domain request queue with configurable delays (e.g., minimum 1 second between requests to the same domain by default). Explain using token bucket or leaky bucket algorithms to smooth out request rates rather than bursty behavior.
Address how rate limits must be enforced globally across multiple crawler workers using a shared store like Redis, ensuring no single domain is hammered even when crawling at scale. Discuss using atomic operations or distributed locks to safely decrement and check rate limit counters.
Explain implementing exponential backoff when receiving 429 (Too Many Requests) or 503 responses, and honoring the Retry-After header when provided. Discuss temporarily deprioritizing or pausing a domain's queue rather than dropping URLs entirely.
Highlight the importance of tracking per-domain metrics like error rates, response times, and crawl throughput to dynamically adjust rate limits. Mention alerting on anomalies such as a sudden spike in 4xx/5xx responses from a specific domain as a signal to back off further.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This was the hardest part and I did not have a clean answer ready.
Frame your answer around the core termination condition problem in distributed crawling: a job is complete when the frontier (queue of URLs to visit) is empty AND no worker is actively processing a URL that could produce new links. Walk through concrete data structures and coordination mechanisms that enforce this guarantee, then address edge cases like cycles and transient failures.
Pro tip: Mention the subtle race condition where the queue appears empty but in-flight workers haven't yet enqueued their discovered links — interviewers at Atlassian love candidates who proactively identify distributed systems pitfalls rather than waiting to be prompted.
Start by clarifying that completion means both the URL frontier is empty AND all in-flight crawl tasks have finished without producing new URLs. This dual condition is the crux of the problem and sets up the rest of your answer.
Describe maintaining three sets or states for URLs: discovered (queued), in-progress (being fetched), and completed (processed). A distributed set like Redis or a database table prevents re-crawling and provides a global view of progress.
Introduce an atomic counter (or distributed semaphore) that increments when a worker picks up a URL and decrements when it finishes — including after enqueuing any new child links. Completion is signaled only when the frontier is empty AND this counter reaches zero.
Explain that without deduplication, cycles in the web graph cause infinite crawling; the visited set acts as a bloom filter or hash set to skip already-seen URLs. Mention canonicalization (normalizing URLs) to avoid treating the same page as distinct.
Discuss how crashed workers can leave URLs stuck in 'in-progress' state, requiring a heartbeat or lease mechanism with a timeout to reclaim and re-queue those URLs. This ensures the termination condition is eventually reachable even under partial failures.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Covered exponential backoff with a max retry cap, dead-letter queue for URLs that keep failing, and per-request timeouts to avoid a single slow page blocking a worker.
Frame your answer around building a resilient, fault-tolerant crawler by addressing failures at multiple layers: network, parsing, and infrastructure. Walk through a concrete strategy covering detection, retry policies, and graceful degradation, demonstrating that you think about both the happy path and edge cases. Tie your decisions back to trade-offs like throughput vs. reliability and resource consumption vs. completeness.
Pro tip: Mention idempotency as a first-class design concern — ensuring that retrying a URL never causes duplicate processing or data corruption signals senior-level thinking and is especially valued at Atlassian where data integrity in tools like Confluence and Jira is critical.
Distinguish between transient failures (network timeouts, 503s), permanent failures (404s, malformed HTML), and systemic failures (target site is down). Each category warrants a different response strategy rather than a one-size-fits-all retry.
Apply exponential backoff with jitter for transient errors to avoid thundering herd problems, and set a maximum retry count (e.g., 3-5 attempts) before marking a URL as failed. Permanent errors like 404 should be logged and skipped immediately without retries.
Set layered timeouts — connection timeout, read timeout, and an overall request deadline — to prevent slow or unresponsive hosts from blocking crawler threads. Use circuit breakers to temporarily stop crawling domains that consistently fail, protecting system resources.
Store failed URLs with metadata (error type, retry count, last attempt timestamp) in a durable queue or database so the system can resume after crashes without losing state. This enables scheduled re-crawls and post-mortem analysis of failure patterns.
Instrument the crawler with metrics on failure rates, retry counts, and timeout frequency per domain to detect anomalies early. Use these signals to dynamically adjust crawl rate limits and timeout thresholds, and set up alerts for sustained failure spikes.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Said I'd persist the queue state and per-URL status to a durable store rather than keeping it only in memory.
Frame your answer around fault tolerance and idempotency: explain how you would checkpoint crawl progress in a durable external store so any worker can resume from the last known good state. Structure your response by first identifying what state needs to be persisted, then how it's stored, and finally how recovery is triggered and executed.
Pro tip: Mention that all state transitions should be idempotent — re-processing a URL that was already crawled should be a no-op — because this eliminates the need for complex exactly-once guarantees and makes recovery trivially safe. This signals production-level thinking that interviewers at Atlassian will appreciate.
Identify all meaningful intermediate state: the URL frontier (pending, in-progress, completed), crawl depth, metadata fetched so far, and any pagination cursors. Being explicit about what constitutes 'state' shows you understand the problem scope.
Select an appropriate storage backend — e.g., Redis with persistence (RDB/AOF), a relational DB, or a distributed KV store like DynamoDB — based on latency, durability, and scale requirements. Justify your choice by weighing consistency needs against performance.
Implement periodic checkpoints where workers atomically write their progress (e.g., last processed URL, batch offset, or page token) before moving to the next unit of work. Use a heartbeat or lease mechanism so the coordinator can detect stale workers.
When a worker crashes or its lease expires, a coordinator (or another worker) reads the last checkpoint and re-enqueues the in-progress work. Ensure all crawl operations are idempotent so re-processing doesn't cause duplicate data or side effects.
Discuss trade-offs such as checkpoint frequency (too frequent adds overhead; too infrequent increases re-work on failure) and how to handle partial writes using atomic operations or two-phase commits. Mention monitoring and alerting for stuck or repeatedly failing jobs.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.