What it does
Growpad customers connect their own website to see how search engines and AI assistants read it. GrowpadBot reads that site’s robots.txt, sitemap and llms.txt, and, for customers who have not connected their own crawl provider, its pages: titles, page summaries, headings and structured data. It only visits sites a Growpad customer has added as their own.
How it identifies itself
User agent:
Mozilla/5.0 (compatible; GrowpadBot/1.0; +https://app.growpad.co/bot)
Requests are signed with Web Bot Auth (HTTP Message Signatures, RFC 9421). The public key is published at https://app.growpad.co/.well-known/http-message-signatures-directory, so you can confirm a request really came from us. A request that claims to be GrowpadBot without a valid signature is not ours.
How it behaves
- It reads and obeys robots.txt, including Crawl-delay.
- If robots.txt answers with a server error or cannot be reached, it does not crawl the site that time.
- It waits at least one second between requests to the same site.
- It fetches a small set of pages per check (25 by default), not the whole site.
- It does not run scripts, submit forms, log in, or try to get past bot challenges.
- When someone adds their own website to Growpad, it reads that home page once to suggest their company details. That one request is made on their behalf.
How to limit or block it
Add this to your robots.txt to block it entirely:
User-agent: GrowpadBot Disallow: /
Or disallow only some paths, or set a Crawl-delay for it. Changes are read at the start of each crawl.
Contact
Questions about GrowpadBot, or a request to stop crawling your site: privacy@growpad.co.