Skip to content

Start Full Site Crawl

POST
/settings/scrape-config/crawl

Crawl a whole site. RUNS IN THE BACKGROUND and can take a long time.

Returns a job id immediately; poll GET /settings/scrape-config/crawl/status/{job_id} for progress. A full crawl fetches every reachable page, so point it at a site you own.

Authorizations

Parameters

Header Parameters

X-Target-Org-Id
Any of:
string

Request Body required

Crawl a whole site. Long-running, and it fetches every reachable page.

object
max_pages
Any of:
integer
>= 10 <= 5000
source_url
Any of:
string

Responses

202

Successful Response

object
data
required

An open envelope for a settings object.

Open because an ontology concept and an SSO config both grow fields as the product does, and on a settings read a dropped field reads as “not configured” rather than “we did not tell you”.

object
key
additional properties
any
pagination
Any of:
object
has_next
required

Whether there are more pages

boolean
limit
required

Items per page

integer
next_cursor
Any of:
string
notes
Any of:
Array<string>
page
required

Current page number (1-indexed)

integer
total
required

Total number of items

integer
total_available
Any of:
integer
total_is_lower_bound
Any of:
boolean

422

Validation Error

object
detail
Array<object>
object
loc
required
Array
msg
required
string
type
required
string