Search Engine Crawler IP Checker

Start with one crawler request in your log

A request from 66.249.66.1 carrying a Googlebot User-Agent only tells you what the client claims to be. The header is editable and is not proof of origin. This page checks one source IPv4 or IPv6 address and separates DNS evidence from published crawler ranges, which is useful when reviewing bot traffic, unexpected crawl volume, or a WAF rule that may be blocking a legitimate crawler.

Copy the address that actually connected to your server. Do not enter a User-Agent, hostname, URL, HTTP status, or an address copied from an unrelated article. If the same source requests hundreds of pages, verify its identity here and then return to the log to review rate, paths, response codes, and robots.txt behavior.

The three evidence groups in the result

The first group is reverse DNS, also called a PTR lookup. The server requests hostnames for the submitted IP and checks recognized suffixes for Google, Bing, Baidu, and Yandex. Examples include googlebot.com for common Google crawlers and search.msn.com for Bingbot.

The second group is forward confirmation. A recognized PTR hostname is resolved again, and the returned addresses must include the original IP. This protects against relying on a misleading or incorrectly configured reverse hostname. A familiar-looking PTR record without a matching forward result is not a complete verification chain.

The third group compares the address with published ranges that were successfully loaded for Google common crawlers, Bingbot, and DuckDuckBot. Baidu and Yandex are currently checked through recognized hostnames and forward DNS, not through a loaded range list. The report keeps those capabilities separate so that a DNS match is not presented as an official range match.

How to run a check on the page

1. Copy one source address from an Nginx, Apache, CDN, load balancer, or application access log. Remove a port, square brackets, and surrounding spaces.
2. Paste it into “IP address to verify.” A valid example is 66.249.66.1; IPv6 may use its standard compressed notation.
3. Select “Verify IP” and wait while the server performs DNS queries and loads published ranges. Repeated clicks will not improve a slow upstream response.
4. Read the overall Verified, Suspicious, or Unverified verdict, then inspect Reverse DNS, Forward DNS confirmation, and Official IP range in order.
5. If a checked source was unavailable, treat that as missing evidence for this run. Do not turn a failed upstream fetch into a negative ownership claim.

The sample buttons demonstrate the interface. For an incident review, always use the address recorded during the relevant time window and retain the matching log lines.

Worked examples and a useful contrast

Possible Googlebot example: enter 66.249.66.1. If it matches the Google common crawlers list that loaded for the request, the verdict will be Verified and the range card will identify Google. A googlebot.com PTR name that resolves forward to the original address adds a second supporting chain. Ranges and DNS records can change, so the live result takes precedence over this example.

Public DNS contrast: enter 8.8.8.8. The address is associated with a Google service, but that does not make it a Google common crawler. If it matches no supported crawler range and produces no recognized forward-confirmed crawler hostname, the result is Unverified. Corporate ownership and crawler identity answer different questions.

Suspicious example: suppose an address returns a PTR name ending in googlebot.com, but resolving that hostname does not return the original IP. The page marks the evidence as suspicious. Do not allow it merely because the name looks familiar, and do not call it malicious without checking the ASN, request paths, timing, and repeated behavior.

Verdict reference

Verified means that at least one identity chain passed at query time. It does not guarantee safe behavior or compliance with your crawl policy. Unverified means that the available checks did not establish a supported identity; it is not a malware or abuse verdict.

CDN and reverse proxy address pitfalls

Behind Cloudflare, another CDN, a load balancer, or your own reverse proxy, the socket address in the web server log may belong to the proxy. Checking that address says nothing about the original visitor. Confirm which header carries the client address and trust it only when a configured proxy inserted it. A client-supplied X-Forwarded-For header must not be accepted without a trusted-proxy policy.

A forwarding chain can contain several addresses, and the correct position depends on your server configuration. An IPv6 address may also appear inside brackets with a port attached. Remove transport notation before submitting it. If the logging layer never captured a reliable source address, a later crawler lookup cannot recover that missing evidence.

Common mistakes and limits

• Trusting User-Agent alone: any HTTP client can copy Googlebot, Bingbot, or Baiduspider text.
• Trusting PTR alone: a recognized reverse name still needs to resolve forward to the original IP.
• Creating a permanent allowlist: crawler networks and DNS records change, so entries need periodic review and narrowly scoped access.
• Treating Unverified as an attack verdict: it only reports insufficient supported evidence. Rate, paths, ASN, errors, and history still matter.
• Using identity to infer indexing: this check cannot tell whether a page was rendered, indexed, ranked, or removed from search.

Even a verified crawler can consume excessive resources or request URLs you do not want crawled. Caching, crawl controls, and proportionate rate limits may still be appropriate. Identity evidence should inform a security decision, not replace a behavior-based policy.

Frequently asked questions

Why can the same IP return a different result later? DNS records, published ranges, and upstream availability change. A timeout can also remove one evidence source from a single run, so record the query time and repeat the check when the decision is important.

Does this identify every search engine crawler? No. Published range matching currently covers Google common crawlers, Bingbot, and DuckDuckBot. Hostname verification also recognizes known suffixes for Baidu and Yandex. Other engines, SEO services, and private bots may remain unverified.

Why does Baiduspider have no official range match? The current implementation does not load a Baidu range document. Baidu-related evidence comes from PTR hostname recognition and forward confirmation, and the page does not label that as a published-range match.

Should every unverified address be blocked? First rule out an incorrect proxy address, DNS timeout, or unavailable list. A high-risk pattern may justify a temporary rate limit or challenge, but a permanent block should be based on sustained evidence.

Can I paste a complete log file? No. The page checks one IPv4 or IPv6 address at a time. A batch review should deduplicate addresses in an authorized logging workflow, preserve timestamps, and control query rate.

You can now return to the input above, submit one real source address from your access log, and save the three evidence cards with your incident notes.