We use cookies.This website uses essential cookies to operate core features. With your consent, we also use analytics cookies to understand traffic and improve the service. For more details, see our .
We use cookies.This website uses essential cookies to operate core features. With your consent, we also use analytics cookies to understand traffic and improve the service. For more details, see our .
Was this tool helpful to use?
Your feedback helps us make it better
Simulate search engine spiders to crawl web pages. Check HTTP crawl status, response headers, and view the raw HTML page source online.
This tool uses the selected search engine bot's User-Agent to request the target page and shows you what the bot actually sees.
Overview
Understand what the tool solves, how it works, and the boundaries of its data.
Enter a URL, select a crawler profile, and request the page to inspect the HTTP response returned for that single fetch. The results include the status code, final URL, response time, content type, selected headers, title and meta fields, heading outline, internal and external links, and returned HTML. This is useful for a quick response-level check when comparing how a server responds to different User-Agent strings.
A User-Agent is text sent with an HTTP request to identify the client. Selecting a Googlebot profile changes the request label; it does not make the request originate from Google. Google notes that crawler User-Agent strings can be spoofed and describes IP or reverse-DNS checks to verify a real Google request. This simulator reports what its own fetch received, not a confirmed visit by a search engine.
Guide
Follow the workflow and verify inputs and outputs with practical examples.
Enter the complete page URL you want to inspect, including its scheme such as https://.
Select a crawler profile from the available list, then run the request. The chosen profile controls the User-Agent sent for the fetch.
Start with the status code, final URL, content type, elapsed time, and response size. Then review the metadata, headings, links, headers, and source tabs for the detail relevant to your check.
If the response is marked truncated, treat the displayed HTML and extracted details as a partial view. Compare with your own browser or server logs when diagnosing a discrepancy.
For example, a developer investigating a missing page title can inspect the returned metadata and HTML to see whether the initial response includes those elements. A response-level snapshot is a diagnostic clue, not an indexing verdict.
Use cases
See how the tool fits into real work and everyday tasks.
Check whether a test page returns the expected status, title, description, headings, and canonical field in the fetched response before publishing a change.
Inspect the final URL and response headers alongside the extracted fields to spot a redirect or a missing element. Follow up in the site’s own logs if the cause is not visible in this snapshot.
Run the same public page with different listed User-Agent profiles and compare the returned response. This can reveal server-side variation, but it does not authenticate any named crawler.
Q&A
Find concise answers to common questions and confusing cases.
No. It sends a Googlebot-like User-Agent with this tool’s request. The label can be spoofed and the request is not verified as Google traffic.
No. An HTTP response describes this fetch. Search indexing is a separate process and cannot be confirmed by the response simulator.
The server may return partial content, or the result may be marked truncated. The tool inspects the received response and does not run the page’s JavaScript to reproduce a browser-rendered page.
Notes
Review scope, result limitations, and important precautions before use.
The selected profile is a request label, not proof of crawler identity. A single fetch cannot establish indexing, ranking, full site crawlability, or what a JavaScript-enabled browser ultimately renders. Results can vary with redirects, access controls, server state, and response truncation. Only submit URLs you are authorized to inspect, and use your server logs and the search engine’s own verification tools for decisions that depend on authentic crawler traffic.
Related
Discover related tools, collections, and available API capabilities.