KutayGO-ScanBot — who we are, and how to let us in
This is the address the crawler's own user agent points at. If a request in your logs saidKutayGO-ScanBot, this page says what it was doing, and what to write if you want it allowed or stopped.
What this crawler is
KutayGO-ScanBot reads a small number of pages on a domainwhose owner asked us to, in order to build a profile of that company: what it does, what it sells, where it is, who runs it — twenty-one fields in all. That profile is then compared against what AI answer engines believe about the same company.
It is not a search index, not a training-data collector, and not an SEO auditor. Nothing it reads is used for any customer other than the one who owns the domain.
- User agent
- KutayGO-ScanBot/1.0 (+https://kutaygo.com/bot; ws=<workspace id>)
- Pages per scan
- 30 at most
- Rate
- 1 request per second, 2 concurrent per host
- robots.txt
- Read first and obeyed, including Crawl-delay
- JavaScript
- Not executed. Static HTML only
- Frequency
- Up to 7 scans per domain per month
- Scope
- Domains a customer has verified they control, and pages that appeared as a competitor in that customer's search results
The ws= value is a workspace identifier. It isnot a credential — it says which of our customers this scan belongs to, so it appears in your logs and you can tell one scan from another.
If we crawled you and you are not a customer
We also read pages that a search engine returned as a competitor for one of our customers, so that the customer's report can say how they compare. That is the only reason we would be on your site without you having asked.
It is one page — the URL that ranked, not a crawl of your site — and only when a cached copy was unavailable. robots.txt is read and obeyed exactly as it is everywhere else, so a file that does not permit us is the end of it: your page is simply recorded as unread, and the report says so.
Mail [email protected] to be excluded entirely and we will stop, whether or notrobots.txt says so.
How to let us in
Which one you need depends on how you are blocking us. If you are not sure, run the check on your workspace's domain page and it will tell you.
1. If your robots.txt denies everything and allows a list
This is the only case a firewall rule cannot fix, becausewe never send the request — we read your file, see no stanza for us, and stop.
User-agent: KutayGO-ScanBot
Allow: /If you already maintain a list of permitted agents (GPTBot,PerplexityBot,ClaudeBot and so on), we belong in the same block. The engines on that list are the ones we are measuring for you; leaving us off it means we report that they know less about you than they actually do.
2. If your edge blocks or challenges us
Two ways, and the first is the one we recommend.
By address and workspace id — nothing secret is sent.
Our requests come from a published address, listed on this page and in/bot/ips.json. Allow requests that matchboth the address and thews= value in the user agent.
The crawler has no assigned egress address yet, so there is nothing for this page or/bot/ips.json to list, and neither of them lists one. We would rather leave the section empty than print an address you would allow on our word. Until it is published, use the token rule below, or mail[email protected] and we will tell you where the requests come from.
Cloudflare, as a WAF custom rule with action Skip:
(ip.src in $kutaygo_scanbot) and (http.user_agent contains "KutayGO-ScanBot")Match on both. The user agent alone is spoofable by anyone; the pair is not.
By token — when you cannot pin our address.
If you sit behind something that rewrites the source address, or you would rather not maintain an IP rule, generate a token for the workspace and we will send it on every request:
X-KutayGO-Access: <token>(http.request.headers["x-kutaygo-access"][0] eq "<token>")Understand what this is before choosing it: the token bypasses your bot protection on your zone. Your access logs may record it. If it leaks, whoever holds it can walk past that protection on your site. We reduce the exposure as far as we can — it is sent only over HTTPS, only to your verified domain and its subdomains (never to a redirect that leaves them), it is scoped to one workspace, and you can rotate it at any time — but the residual risk is yours and it is the reason the address-based rule is the default.
If you would rather not let us in at all
That is a legitimate choice and the product still works. You fill in your company's details yourself, we still measure what the answer engines believe, and we still score the difference.
The one thing that changes is what the report says about itself: fields you entered by hand are markedself-declared rather than read from your site, and the report states this on its face. Anyone you show the score to should be able to tell which kind of ground truth it was checked against.
[email protected]
An operator who wants us to stop, slow down, or explain a request reaches a human there. A crawler with no way to complain about it is a crawler that gets blocked instead of contacted.
We will stop crawling a domain on the owner's request, whether or not they are a customer, and whether or notrobots.txt says so.