Crawler information
CiteluraBot
CiteluraBot is the crawler behind Citelura audits. It reads public pages to check technical SEO and how ready a site is for AI answer engines.
Live audits are not switched on yet, so CiteluraBot is not crawling any website at the moment.
Who runs it
CiteluraBot is operated by Citelura. It runs only when someone asks Citelura to audit a website, and it identifies itself in every request.
How it identifies itself
Every request carries this user agent. The address in brackets is this site.
CiteluraBot/0.1 (+https://citelura.com/bot; SEO and AI answer-engine readiness audit; respects robots.txt)
What it reads
- robots.txt first, on every site it visits.
- The sitemap, when the site lists one.
- A small, bounded number of public pages on the same site: up to 11 pages on a free audit, and never more than 50 on any plan.
Its limits
- Up to 3 redirects per request, and every redirect is checked again.
- Pages up to 2048 KiB, and robots.txt up to 500 KiB.
- 10 seconds per request and 25 seconds in total per page.
- At most 2 requests at the same time.
What it never does
- Contact private, internal or non-public addresses.
- Log in, fill in forms or accept cookies.
- Follow links to other websites.
- Store raw page HTML. Pages are analysed in memory.
How to block it
Add this to the robots.txt file at the root of your site. CiteluraBot reads robots.txt before it fetches anything else, and it does not crawl a site whose robots.txt it cannot read.
robots.txt snippet
User-agent: CiteluraBot Disallow: /
Contact
Questions, or a problem with blocking? Write to bot@citelura.com.