CrescontentBot
CrescontentBot is the web crawler used by Crescontent, a service that helps website owners write help pages for their own websites. It identifies itself with the user agent:
CrescontentBot/1.0 (+https://crescontent.com/bot)
When it visits
Only when the owner of a website has added it to Crescontent, proved they control it (with a meta tag or DNS record), and asked us to draft help topics. It does not crawl the web generally.
What it does
- Reads up to 25 publicly available pages on that one website, preferring help-like pages such as FAQs, pricing and contact pages.
- Follows the website's
robots.txtrules, and skips login, account, cart and checkout pages. - Sends the page text to our AI provider to draft help topics for the owner to review. The page text is not kept.
- Never fills in forms, signs in, or follows links to other websites.
Blocking it
To stop it reading your site, add this to your robots.txt:
User-agent: CrescontentBot
Disallow: /
Contact
Questions or concerns: info@crescodigo.com. Crescontent is operated by Crescodigo Limited (company number 17569621). See our Privacy Notice.