The /crawl endpoint scrapes content from a starting URL and follows links across the site, up to a configurable depth or page limit. It respects robots.txt and AI Crawl Control by default. Crawl jobs run asynchronously: you submit a URL, receive a job ID, and check back for results. Great for building knowledge bases, RAG pipelines, and content monitoring.