Export gives you raw crawler visit data for analysis in spreadsheets, BI tools, or custom scripts. Each row represents a single crawler visit with the path, bot name, timestamp, and data source.
Formats#
Two formats are available:
- CSV — for spreadsheets and data tools. Includes a header row (
path,bot,visited_at,source). Fields containing commas, quotes, or newlines are properly escaped. - JSON — for programmatic use. Returns an array of visit objects with the same four fields.
Constraints#
- 10,000 rows maximum per export. Rows are ordered oldest-first. If your site has more visits than the limit in the selected window, reduce the date range to narrow the result set.
- 1 export per 5 minutes per user. Exceeding this returns a rate limit error.
- Up to 90 days of data available.
Example#
curl -H "Authorization: Bearer $API_KEY" \
"https://www.crawlready.app/api/v1/analytics/$SITE_ID/export?format=csv&days=30" \
-o crawlready-export.csv
API access#
See the Export endpoint in the API Reference for the full specification.
Frequently Asked Questions
What formats can I export crawler data in?
Two formats: CSV, which includes a header row (path, bot, visited_at, source) and properly escapes fields containing commas, quotes, or newlines, and JSON, which returns an array of visit objects with the same four fields.
What's the maximum number of rows I can export?
10,000 rows per export, ordered oldest-first. If your site has more visits than that in the selected window, narrow the date range to reduce the result set.
Is there a rate limit on exports?
Yes — one export per 5 minutes per user. Exceeding that returns a rate limit error.
How far back does exported data go?
Up to 90 days of crawler visit data is available for export.
