SiteSnapshot crawler

SiteSnapshot/1.0 is the page crawler of this SEO tool. If you found this page through a line in your server logs, the request came from it. Its User-Agent reads SiteSnapshot/1.0 (+https://seostudio.com/docs/site-snapshot), or a browser's carrying that name when it renders a page (see JavaScript). Its robots.txt token is SiteSnapshot.

At a glance

To block it, see How to block it. To let it through a firewall, see Allowing it through a firewall.

Why it visits your site

It visits a site only because someone added that site to a project here, to watch their own pages: what each page says, when it changed, and what is broken. It reads public pages only: it never submits a form or logs in, stays on the site's own host (and its subdomains, when the person who added the site asked for them), and does not follow a link marked nofollow (or any link on a page whose robots meta tag says nofollow).

• When the person who added the site has verified it in Google Search Console or Bing Webmaster Tools, it reads the pages that get search impressions, the URLs in the site's sitemaps and the pages they link to, up to a page limit per site.
• When the site is not verified, it reads at most 500 pages a week, starting from the home page and the sitemaps.

How often

It waits at least 1 second between two requests to the same host, follows at most 5 redirects, gives up on a request after 15 seconds and reads at most 5 MB of a page. The one exception is a large shared file CDN built to serve every site on it (such as cdn.shopify.com, never a site's own server), which it asks at most 8 times a second in all.

It always says who it is

Requests from our servers always carry the SiteSnapshot User-Agent above, never a search engine's, so you can always tell it apart from Googlebot or Bingbot and allow or block it on its own. A person who audits their own site may ask it to follow the robots.txt rules written for Googlebot or Bingbot instead of its own, on a site they verified they own; it still sends its own name.

Our desktop app is different: it runs on the person's own computer, from their own address, and sends the user agent they choose there, the way desktop SEO crawlers do. Those requests do not come from our addresses and carry no signature.

Verifying a request

Every request from our servers is signed with Web Bot Auth (HTTP Message Signatures, RFC 9421, Ed25519). A signed request carries three headers:

• Signature-Agent: the site whose key directory holds our public key, in double quotes;
• Signature-Input: the signed parts (@authority and signature-agent), when the signature was made and when it expires, the key id and tag="web-bot-auth";
• Signature: the signature.

The public key is served at /.well-known/http-message-signatures-directory on the site the Signature-Agent header names; for this site that is the Web Bot Auth key link in At a glance. Cloudflare and other services that support Web Bot Auth verify these signatures for you. A User-Agent can be copied, a signature cannot, so a signed request is the surest proof a visit came from us.

Our addresses

Allowing it through a firewall

If a firewall or bot protection answers the crawler with a challenge, a 403 or a 429, the audit cannot read your pages. Allow it by its user agent, and where the tool supports it also by our addresses.

Cloudflare

1. Open your domain, go to Security > Security rules and create a custom rule.
2. Set the field to User Agent, the operator to contains and the value to SiteSnapshot.
3. Choose the action Skip, tick every remaining custom rule, rate limiting rule, managed rule and Super Bot Fight Mode, then deploy.
4. On the free plan, turn off Bot Fight Mode in the security settings: no rule can skip it.

Sucuri

1. In the Sucuri firewall dashboard, open Access Control.
2. Under the allowlist of IP addresses, add each of our addresses.
3. Save. Sucuri cannot allow a user agent, so it needs our addresses.

Wordfence

1. In WordPress, open Wordfence > All Options.
2. Under Advanced Firewall Options, add each of our addresses to the allowlisted IP addresses that bypass all rules.
3. Save the changes.

Any other firewall or host

1. Allow requests whose user agent contains SiteSnapshot.
2. Where the tool allows it, also limit that rule to our addresses, so nobody else can pass as our crawler.
3. If you cannot change these settings, send this page to your host and ask them to allow the crawler.

JavaScript

Like Google, it renders each page it reads in a browser, so it sees what JavaScript adds. The browser renders the HTML the crawler already fetched (the page is not requested twice) and loads the page's scripts, stylesheets and data requests, never its images, fonts or media. It waits until the page settles, for 15 seconds at most. The page itself and those requests carry a phone browser's User-Agent with the crawler's name and the word bot in it, so a site that serves phones its own layout is read as a phone gets it, and a site that serves crawlers a page built on its server is read as search engine crawlers get it:

Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version>.0.0.0 Mobile Safari/537.36 (compatible; SiteSnapshot/1.0; bot; +https://seostudio.com/docs/site-snapshot)

A render of the desktop layout (when the person who added the site asked for it) carries a desktop browser's User-Agent with the same name instead:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<version>.0.0.0 Safari/537.36 (compatible; SiteSnapshot/1.0; bot; +https://seostudio.com/docs/site-snapshot)

It keeps a site's scripts and stylesheets for up to ten minutes, so the same file is not fetched for every page. It never sends a form, never clicks, and never keeps a cookie from one page to the next.

Files and outbound links

To find broken, redirected and oversized files, it checks the images, stylesheets, scripts and fonts a site's pages load, once per file however many pages use it, at most once a week. It asks with a HEAD request and, only when that does not tell the file's status or size, a GET for its first byte. A few files get one more GET: the first 32 KB of an image of some weight (where its pixel size is written), a text file that named no compression asked again offering gzip and brotli (its body not read), a compressed script or stylesheet read once for its unpacked size, and the first and last bytes of a PDF (where its title is written). It checks the links a site's pages make to other sites the same way, with a HEAD request (a GET only when the HEAD is refused), at most once every 30 days per link. These requests carry the crawler's own User-Agent, keep the same pause between two requests to one host (shorter only on a large shared file CDN, as above), and on the site itself obey its robots.txt.

When a site names language versions of its pages on other domains (hreflang), the crawler reads one such page on each of at most 10 of those domains per crawl, after that domain's robots.txt, to see whether it names the site's page back. It reads no other page there.

robots.txt

It reads your robots.txt before anything else and obeys it the way RFC 9309 describes: a group for SiteSnapshot wins over the * group, the longest matching rule wins, and Allow wins a tie. A Crawl-delay in the group it follows is honoured, up to 10 seconds between requests. A page that robots.txt disallows is never fetched, and neither is a script, stylesheet or data file on your site that it disallows when a page is rendered.

How to block it

Add this to your robots.txt to block it from the whole site:

User-agent: SiteSnapshot
Disallow: /

Or block only part of the site, for example a folder:

User-agent: SiteSnapshot
Disallow: /private/

To slow it down instead, add a delay in seconds:

User-agent: SiteSnapshot
Crawl-delay: 5

The change applies at its next visit, as soon as it reads the file again.

Contact

Questions, or a crawl you did not expect? Write to [email protected].