Agentic Crawler APIs

Agentic Crawler APIs start AI-driven web crawling workflows that discover and collect structured results from the open web based on a natural-language prompt.

Overview

The Agentic Crawler uses a two-step flow: the start endpoint prepares a session and returns a WebSocket URL; the agent workflow does not run until you connect to that WebSocket with your API key.

Important: Calling POST /agentic-crawler/start alone does not start the crawl. You must open the returned websocket_url, replace the <your-api-key> placeholder with your real API key, and successfully connect. The agent begins processing only after the WebSocket connection is established.

The workflow includes:

Start – Submit a natural-language prompt describing what to find. The API returns a thread_id, list_id, and websocket_url with status ready.
Connect WebSocket – Replace <your-api-key> in websocket_url with your API key and open the connection. This step triggers the agent.
Clarifying answers – Send follow-up answers over WebSocket using prompt, thread_id, and list_id.
Stream progress – Receive real-time crawl updates and results over the WebSocket.

Base URL: https://chordian-core.chordian.ai

Authentication: Bearer token. Send your API key in the header: Authorization: Bearer <api-key>.

Start Agentic Crawler

Prepare an agentic crawler session for the given prompt. This endpoint registers the workflow and returns a WebSocket URL — it does not start the agent until you connect to that URL.

POST /agentic-crawler/start

Authentication: Authorization: Bearer <api-key>

Request Body

{
  "prompt": "Find product managers at mid-stage fintech startups in London"
}

Field	Type	Required	Description
`prompt`	string	✅	Natural-language description of the people, companies, or data to find.

Successful Response (200 OK)

{
  "success": true,
  "message": "Connect to websocket_url to start the search",
  "thread_id": "536c****-****-45c5-****-0a8dfa5*****",
  "list_id": "6a3b60****2ea3a0609******",
  "websocket_url": "wss://agent.chordian.ai/api/agent/agentic-crawler?api-key=<your-api-key>&thread_id=536c****-****-45c5-****-0a8dfa5*****",
  "status": "ready"
}

Field	Type	Description
`success`	boolean	`true` when the session was prepared successfully. The agent has not started yet.
`message`	string	Human-readable instruction to connect to the WebSocket to trigger the workflow.
`thread_id`	string	Workflow thread identifier. Included in the WebSocket URL.
`list_id`	string	Identifier for the list that will store crawl results.
`websocket_url`	string	WebSocket endpoint. Replace `<your-api-key>` with your API key and connect to start the agent.
`status`	string	Session state before connection (e.g. `ready` — waiting for a WebSocket client to connect).

Important: The status value ready means the session is prepared but the crawl has not begun. Connect to websocket_url (with your API key substituted for <your-api-key>) to trigger the agent.

Validation Error (422)

{
  "detail": [
    {
      "loc": ["string", 0],
      "msg": "string",
      "type": "string"
    }
  ]
}

Example

curl -X POST https://chordian-core.chordian.ai/agentic-crawler/start \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <api-key>" \
  -d '{
    "prompt": "Find product managers at mid-stage fintech startups in London"
  }'

Connect to WebSocket

This step triggers the agent. After a successful start response, open the websocket_url with your API key in place of the <your-api-key> placeholder. The crawl begins only when the WebSocket connection is successfully established.

The URL format is:

wss://agent.chordian.ai/api/agent/agentic-crawler?api-key=<your-api-key>&thread_id=<thread_id>

Take websocket_url from the start response.
Replace <your-api-key> with your actual API key.
Open the WebSocket connection — the agent starts on successful connect.
Listen for messages to receive progress and results.

If you do not connect, or if you leave the placeholder unchanged, the workflow will not run.

Clarifying Question Payload (WebSocket)

For clarifying-question answers, send this JSON shape over the connected WebSocket:

{
  "prompt": "Limit results to companies founded after 2018",
  "thread_id": "536c****-****-45c5-****-0a8dfa5*****",
  "list_id": "6a3b60****2ea3a0609******"
}

JavaScript Example

const response = await fetch('https://chordian-core.chordian.ai/agentic-crawler/start', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer <api-key>'
  },
  body: JSON.stringify({
    prompt: 'Find product managers at mid-stage fintech startups in London'
  })
});
 
const data = await response.json();
const ws = new WebSocket(data.websocket_url.replace('<your-api-key>', '<api-key>'));
 
ws.onmessage = (event) => {
  console.log('Crawler update:', JSON.parse(event.data));
};
 
ws.onopen = () => console.log('Connected — agent workflow is now running');
ws.onerror = (error) => console.error('WebSocket error:', error);

Next Steps

Explore Company Search APIs for structured company list building
Explore People Search APIs for contact discovery workflows
See Chordian Deep Search APIs for deep research with file uploads

Deep Research APIs Enterprise Search APIs