Optional
allowOptional
categories?: undefined | CrawlCategory[]Only crawl URLs containing these categories.
Optional
excludeExclude these domains.
Optional
excludeExclude these paths.
Optional
extractThe depth of the extractions. It can be "basic" or "advanced".
Optional
format?: undefined | "markdown" | "text"The format of the respose. It can be "markdown" or "text"
Optional
includeInclude images in the crawl results.
Optional
include_Whether to include the favicon URL for each result.
Optional
instructions?: undefined | stringNatural language instructions to guide the crawler
Optional
limit?: undefined | numberThe maximum number of pages to crawl.
Optional
maxThe maximum number of pages to crawl per level.
Optional
maxThe maximum number of hops from the starting URL.
Optional
selectOnly crawl these domains.
Optional
selectOnly crawl URLs containing these paths.
The URL to start crawling from.
Example: "https://example.com"
Allow crawling external domains.