Automate Indexing via CI & Webhooks

Once indexing is on, CrawlReady already watches your sitemap and your page content and submits changes on its own. Most sites need nothing on this page.

You want something here if your sitemap is slow to update, if you publish through a system CrawlReady cannot see, or if you want a deploy to trigger submissions the moment it lands rather than on the next sweep.

Everything below ends up in the same pipeline with the same guard rails. There is no fast path that skips the debounce, and nothing you do here can submit an unchanged page twice in a day.

The one-liner#

curl -X POST https://www.crawlready.app/api/v1/index \
  -H "Authorization: Bearer $CRAWLREADY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"urls":["https://example.com/blog/new-post"]}'

No site ID needed. URLs are matched to your sites by hostname, so one call can carry pages from several domains you own. Up to 500 URLs per call.

The response tells you what happened to each URL individually:

{
  "batch_id": "9f1c…",
  "queued": 1, "skipped": 0, "rejected": 0,
  "engines": ["indexnow", "google"],
  "results": [
    { "url": "https://example.com/blog/new-post", "status": "queued" }
  ],
  "quota": { "used": 341, "included": 10000, "period": "2026-08" }
}

A URL you submitted too recently comes back as skipped, not as an error:

{
  "url": "https://example.com/pricing",
  "status": "skipped",
  "reason": "debounced",
  "next_eligible_at": "2026-08-14T09:00:00Z"
}

That is deliberate, and it is why this is safe to run on every push. A CI job that submits your whole sitemap on each build does not gradually poison your standing with search engines — the ones that have not changed are skipped, cost no quota, and tell you when they become eligible.

One bad URL never fails the batch. A hostname you do not own comes back as rejected with a code, and the rest still go out.

From a sitemap#

Regenerated your sitemap and want everything new or changed in it submitted?

curl -X POST https://www.crawlready.app/api/v1/index/sitemap \
  -H "Authorization: Bearer $CRAWLREADY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"sitemap_url":"https://example.com/sitemap.xml"}'

We fetch it, work out what is new or has a newer lastmod, and submit that. Unchanged entries are skipped without consuming anything.

From CI#

GitHub Actions, after a deploy:

- name: Tell search engines about the deploy
  env:
    CRAWLREADY_API_KEY: ${{ secrets.CRAWLREADY_API_KEY }}
  run: |
    curl -sS -X POST https://www.crawlready.app/api/v1/index/sitemap \
      -H "Authorization: Bearer $CRAWLREADY_API_KEY" \
      -H "Content-Type: application/json" \
      -d '{"sitemap_url":"https://example.com/sitemap.xml"}'

Only the pages this commit touched, which is tidier on a large site:

git diff --name-only HEAD^ HEAD -- 'content/**/*.md' \
  | sed 's|content/|https://example.com/|; s|\.md$||' \
  | jq -Rsc '{urls: split("\n") | map(select(length > 0))}' \
  | curl -sS -X POST https://www.crawlready.app/api/v1/index \
      -H "Authorization: Bearer $CRAWLREADY_API_KEY" \
      -H "Content-Type: application/json" -d @-

With the CLI, if you already have it installed:

crawlready index https://example.com/blog/new-post
crawlready index --sitemap https://example.com/sitemap.xml

From a deploy webhook#

If you would rather click than write CI, point your host's deploy notification at CrawlReady. These carry no URL list — they trigger a sitemap re-read, and whatever changed gets submitted. A site with a correct sitemap gets fast indexing having written no code at all.

https://www.crawlready.app/api/webhooks/vercel/{siteId}
https://www.crawlready.app/api/webhooks/netlify/{siteId}
https://www.crawlready.app/api/webhooks/generic/{siteId}

Vercel — add a webhook for the deployment.succeeded event and set the signing secret to your site's webhook secret from Settings. Verified against x-vercel-signature.

Netlify — add an outgoing notification for "Deploy succeeded" with the same secret. Verified against x-webhook-signature.

Generic — HMAC-SHA256 over the raw request body, sent as x-crawlready-signature. Use this for any other host or a custom script.

Only successful deploys trigger anything. Build-started, build-failed and alias events are ignored — running discovery against content that was never published is worse than doing nothing.

From your own application#

If you already call /api/v1/recache when you deploy, you are done: that endpoint now marks changed pages for indexing as well as refreshing the cache. Nothing to change.

What is guarding all of this#

Every route above lands in the same place, and the same rules apply to all of them:

None of these can be switched off, and that is the point. Search engines throttle and eventually ignore hosts that announce changes to pages that did not change, and the damage takes weeks to undo. force: true exists for a human clicking a button; it is still floored at one hour and it still consumes quota.

Decisions#

• One endpoint, no site ID. Hosts are matched against the sites your organization owns, so one integration covers every domain you manage.
• Partial success is the normal case. Per-URL outcomes, never a whole-batch failure because one URL was wrong.
• Skips are reported, not swallowed. A debounced URL returns the time it becomes eligible. Silently accepting and quietly dropping would make CI look like it worked.
• Unknown pages are created, not rejected. Publishing a brand-new page and indexing it has to be one call, since that is the entire point.