Atlassian·Software Engineer·Onsite - System Design / Architecture
- Design a REST API service where clients submit URLs and the service crawls each URL plus all its sublinks to extract image references. The API should support submitting a batch of URLs and polling for job status and results.
- How would you architect the crawler itself? Walk through the worker pool, the URL frontier/queue, and how you'd deduplicate already-visited URLs.
- How would you handle polite crawling, including rate limiting per domain and respecting robots.txt?
- How do you know when a crawl job is fully complete? Specifically, how do you detect that all sublinks have been discovered and processed?
- How would you handle failures, retries, and timeouts in the crawler?
- How would you store intermediate crawl state so the system can recover if a worker crashes mid-job?
“I got the basic shape of it pretty quickly: POST to create a job, GET to poll status, return discovered images once done.”