A redesign is about to ship, a client wants a record of the old site before the migration, or an audit needs every page as an image. Doing it by hand works for five pages. At fifty it is an afternoon of pasting URLs, and at five hundred nobody does it.
You do not need a crawler to find the pages. Almost every site already publishes the list in its sitemap, the file search engines read. The job is three loops: read the sitemap, capture each URL, and write down which captures actually worked. The third step is the one people skip, and it is the one that matters: a screenshot API can hand you a perfectly valid PNG of a "Not Found" page with an HTTP 200, and a folder of files looks the same whether every page rendered or a fifth of them were errors.
One page first, with curl and your key in SITESHOT_API_KEY:
printf '%s' "$SITESHOT_API_KEY" | curl --fail-with-body --silent --show-error \
--get "https://api.site-shot.com/" \
--data-urlencode 'userkey@-' --data-urlencode 'url=https://www.wikipedia.org/' \
--data 'full_size=1' --output page.png
The key travels on standard input, so it stays out of your shell history and curl's arguments; it is still part of the request URL, so keep request logs private. The rest of this guide turns that one request into every page plus a manifest.
The short answer
For a handful of pages, paste each URL into the free browser tool: no signup, one page at a time. For a whole site you need the API. A confirmed account's API key comes with the free plan: 100 API captures a month, no card needed; paid plans start at $5/mo for 2,000. Get the key from the API quickstart, export it as SITESHOT_API_KEY, and run the steps below on three pages before you run them on all of them.
Step 1: Get the list of pages from the sitemap
The authoritative pointer to a sitemap is the Sitemap: line in the site's robots.txt, which is how the sitemaps.org protocol tells crawlers where to look:
curl --silent https://www.example.com/robots.txt | grep -i '^sitemap:'
On our own site that prints Sitemap: https://www.site-shot.com/sitemap.xml. No line? Try /sitemap.xml at the root. Then pull every <loc> out of it:
SITEMAP="https://www.example.com/sitemap.xml" # your site's sitemap
curl --fail --silent --show-error --compressed "$SITEMAP" \
| grep -o '<loc>[^<]*</loc>' \
| sed -e 's/<loc>//' -e 's/<\/loc>//' -e 's/&/\&/g' \
> urls.txt
wc -l < urls.txt
grep -o works even when the whole sitemap sits on one line, as ours does. The sed undoes the & escaping the protocol requires inside URLs. Before you capture anything, check three things:
- Is it a sitemap index? If the file's root element is
<sitemapindex>, its<loc>entries are more sitemaps, not pages. One sitemap file may hold at most 50,000 URLs and 50 MB, so large sites are almost always indexes. Thegrepabove would hand you the child sitemap URLs; the Python reader in step 3 follows them for you, and also opens.xml.gzsitemaps. - Is every URL a page? Site-Shot's own sitemap, checked October 31, 2026, lists 68 URLs, and 18 of them are
.txt,.jsonand.mdfiles written for AI agents. A capture of a text file costs the same as a capture of a page. Filter first. - How many are there?
wc -lis your budget. The sitemap index of docs.djangoproject.com expanded to 32,846 page URLs on October 31, 2026, every language and version of the docs. Knowing that before the first request is the difference between a test run and a surprise.
Step 2: Capture every URL with a curl loop
The simplest batch is a shell loop over urls.txt, one request per page, one page at a time:
mkdir -p shots
n=0
while IFS= read -r url; do
n=$((n + 1))
file="shots/$(printf '%04d' "$n").png"
if printf '%s' "$SITESHOT_API_KEY" | curl --fail-with-body --silent --show-error \
--get "https://api.site-shot.com/" \
--data-urlencode 'userkey@-' --data-urlencode "url=$url" \
--data "full_size=1&max_height=15000&no_cookie_popup=1" \
--output "$file"; then
printf 'ok\t%s\t%s\n' "$file" "$url" >> shots/index.tsv
else
rm -f "$file"
printf 'FAILED\t-\t%s\n' "$url" >> shots/index.tsv
fi
done < urls.txt
full_size=1 captures the whole scrolling page, max_height caps it at 15,000 pixels, and no_cookie_popup=1 removes consent banners. We ran this loop on October 31, 2026 against three URLs: our pricing page, a path that does not exist on our site, and a host that does not exist at all. shots/index.tsv said:
ok shots/0001.png https://www.site-shot.com/pricing/
ok shots/0002.png https://www.site-shot.com/this-page-does-not-exist-404/
FAILED - https://no-such-host-zz91.example-invalid.com/
The missing host failed loudly: the API answered with an HTTP error, curl exited non-zero, the row says FAILED. The second row is the trap. The API answered HTTP 200 with a 1024×768 PNG of the site's "Not Found" page, a real screenshot of the wrong thing. An HTTP 200 alone does not prove a capture succeeded; it proves the API returned an image. To tell the two apart you need the details the image mode does not send.
Step 3: Keep a manifest of what worked
Add response_type=json and the API returns the image as a base64 data URL inside a JSON body, next to fields that say what happened. A quick way to see them, with url set to the page:
printf '%s' "$SITESHOT_API_KEY" | curl --silent --show-error \
--get "https://api.site-shot.com/" \
--data-urlencode 'userkey@-' --data-urlencode "url=$url" \
--data "response_type=json" \
| jq -c '{error, error_code, page_status: .response.status_code, image_chars: (.image | length)}'
For https://example.com/, the missing page and the missing host, on October 31, 2026, every response was an HTTP 200, and the JSON told them apart:
{"error":null,"error_code":null,"page_status":200,"image_chars":113362}
{"error":null,"error_code":null,"page_status":404,"image_chars":13590}
{"error":"504 Gateway Timeout","error_code":"proxy_navigation_failed","page_status":"504","image_chars":22}
Two checks, then. A non-empty error means the capture failed and the image is empty. response.status_code is the status the page itself answered, so a 404 there means you captured an error page.
The official Python SDK (pip install site-shot) asks for JSON on every call and raises SiteShotError when error is set, so the script below adds only the page-status check, the sitemap reader and the bookkeeping. Save it as sitemap_shots.py:
import base64
import csv
import gzip
import sys
import urllib.request
import xml.etree.ElementTree as ET
from datetime import date
from pathlib import Path
from urllib.parse import urlsplit
from site_shot import SiteShot, SiteShotError
NS = "{http://www.sitemaps.org/schemas/sitemap/0.9}"
NOT_PAGES = (".xml", ".txt", ".json", ".md", ".pdf", ".jpg", ".png", ".zip")
VARIANTS = {
"desktop": {"width": 1280, "full_size": True, "max_height": 15000},
}
def read_sitemap(url):
request = urllib.request.Request(url, headers={"User-Agent": "sitemap-screenshots"})
with urllib.request.urlopen(request, timeout=30) as response:
data = response.read()
if data[:2] == b"\x1f\x8b": # a .xml.gz sitemap
data = gzip.decompress(data)
return ET.fromstring(data)
def page_urls(sitemap_url):
root = read_sitemap(sitemap_url)
locs = [loc.text.strip() for loc in root.iter(NS + "loc") if loc.text]
if root.tag == NS + "sitemapindex": # an index lists more sitemaps
for child in locs:
yield from page_urls(child)
else:
yield from (u for u in locs if not urlsplit(u).path.lower().endswith(NOT_PAGES))
def file_name(n, url, variant):
slug = urlsplit(url).path.strip("/").replace("/", "_") or "home"
return f"{n:04d}-{slug[:80]}-{variant}.png"
def main(sitemap_url, limit=None):
client = SiteShot() # reads SITESHOT_API_KEY
out = Path("shots") / date.today().isoformat() # one folder per run
out.mkdir(parents=True, exist_ok=True)
urls = list(dict.fromkeys(page_urls(sitemap_url)))[:limit]
failed = 0
with open(out / "manifest.csv", "w", newline="") as handle:
manifest = csv.writer(handle)
manifest.writerow(["url", "variant", "file", "page_status", "outcome", "detail"])
for n, url in enumerate(urls, 1):
for variant, options in VARIANTS.items():
name, status, outcome, detail = "", "", "ok", ""
try:
result = client.capture_json(url, no_cookie_popup=True, **options)
status = result.get("response", {}).get("status_code", "")
image = result["image"].split(",", 1)[-1]
name = file_name(n, url, variant)
(out / name).write_bytes(base64.b64decode(image))
if not isinstance(status, int) or status >= 400:
outcome, detail = "check", f"page answered {status}"
except SiteShotError as error: # the JSON carried "error"
outcome, detail = "failed", str(error)
if outcome != "ok":
failed += 1
manifest.writerow([url, variant, name, status, outcome, detail])
print(outcome, variant, url)
print(f"{len(urls)} pages, {failed} rows need a look")
return 1 if failed else 0
if __name__ == "__main__":
limit = int(sys.argv[2]) if len(sys.argv) > 2 else None
sys.exit(main(sys.argv[1], limit))
Install the SDK and run it on the first three pages:
python -m pip install site-shot
python sitemap_shots.py https://www.example.com/sitemap.xml 3
Against Site-Shot's own sitemap on October 31, 2026, it printed:
ok desktop https://www.site-shot.com/
ok desktop https://www.site-shot.com/ai-agents/
ok desktop https://www.site-shot.com/api/v1/agent/plans/
3 pages, 0 rows need a look
Look at the third row. It is a JSON endpoint without a file extension, so the filter let it through and the capture "worked": a picture of JSON text. ok means the API produced an image of a page that answered below 400. It does not mean the page was worth capturing, or that the image shows what you wanted.
Fed a sitemap with our API quickstart page, the missing page and the missing host, the manifest kept all three outcomes and the script exited with status 1:
url,variant,file,page_status,outcome,detail
https://www.site-shot.com/start/,desktop,0001-start-desktop.png,200,ok,
https://www.site-shot.com/this-page-does-not-exist-404/,desktop,0002-this-page-does-not-exist-404-desktop.png,404,check,page answered 404
https://no-such-host-zz91.example-invalid.com/,desktop,,,failed,Site-Shot render timed out: 504 Gateway Timeout
The non-zero exit makes a scheduled run fail visibly instead of quietly. Then open the images anyway, at least the check rows and a sample of the ok ones: look for blank pages, sign-in screens, bot challenges, cookie walls the remover missed, full pages clipped at max_height, and error messages painted as pictures. A page that does not finish loading in time can come back as an image of a timeout message rather than as an error. The manifest narrows the search; your eyes finish it. The Python screenshot guide covers the SDK itself.
The same manifest in Node.js
The Node.js SDK (npm install site-shot-sdk) does the same error check inside captureJson(). This version reads the urls.txt from step 1. Save it as shots.mjs and run node shots.mjs:
import { SiteShot, SiteShotError } from "site-shot-sdk";
import fs from "node:fs/promises";
const client = new SiteShot(); // reads SITESHOT_API_KEY
const urls = (await fs.readFile("urls.txt", "utf8")).split("\n").filter(Boolean);
await fs.mkdir("shots", { recursive: true });
const rows = [["url", "file", "page_status", "outcome", "detail"]];
for (const [i, url] of urls.entries()) { // one at a time, in order
const file = `shots/${String(i + 1).padStart(4, "0")}.png`;
try {
const result = await client.captureJson({ url, width: 1280, full_size: true,
max_height: 15000, no_cookie_popup: true });
const status = result.response?.status_code;
await fs.writeFile(file, Buffer.from(result.image.split(",").pop(), "base64"));
rows.push([url, file, status, status < 400 ? "ok" : "check", ""]);
} catch (error) {
if (!(error instanceof SiteShotError)) throw error;
rows.push([url, "", "", "failed", error.message]);
}
console.log(rows.at(-1)[3], url);
}
await fs.writeFile("shots/manifest.json", JSON.stringify(rows, null, 2));
On Node.js 24 with the same three URLs it printed ok, check and failed, and wrote the three rows to shots/manifest.json. See the Node.js screenshot guide for the SDK's other options.
Step 4: Add a mobile pass
Both SDKs pass width and user_agent straight to the API, so a mobile layout is one more entry in the Python script's VARIANTS:
IPHONE = ("Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) "
"AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1")
VARIANTS = {
"desktop": {"width": 1280, "full_size": True, "max_height": 15000},
"mobile": {"width": 375, "height": 812, "full_size": True, "max_height": 15000,
"user_agent": IPHONE},
}
Every page now produces two rows in the manifest and two files, -desktop.png and -mobile.png. On our quickstart page the mobile capture came back 375 pixels wide and 2,776 tall, in the site's phone layout. This is a narrow Chromium viewport with an iPhone user agent, not a real iPhone or Safari session, and every variant is one more capture per page. More on the difference between page and screen captures in full page vs viewport screenshots.
How many captures will this cost?
Pages × variants × runs. Test runs count, and so does every retry of a failed row.
| Site | Variants | Captures per run |
|---|---|---|
| 40 pages | desktop | 40 |
| 40 pages | desktop + mobile | 80 |
| 150 pages | desktop + mobile | 300 |
| 1,000 pages | desktop + mobile | 2,000 |
A confirmed account's API key comes with the free plan: 100 API captures a month, no card needed; paid plans start at $5/mo for 2,000. The first two rows fit inside a month of the free plan. The 150-page site in two layouts does not, and once a month it fits easily on the cheapest paid plan. Past a paid plan's allowance, each extra capture is billed at the plan's rate, $0.0025 on the $5 plan (pricing, checked October 31, 2026). If the month's captures run out mid-run, the remaining rows land in the manifest as failed with the API's quota message, and the exit code says so.
Re-run it every month
A month-on-month archive is the same script on a timer. Use cron on a machine that is on at that hour (a server, not a laptop that sleeps). Cron does not see a virtual environment you activated in your shell, so give the job its own: put sitemap_shots.py in one folder and install the SDK into a virtual environment there.
mkdir -p "$HOME/site-shots" && cd "$HOME/site-shots"
python3 -m venv .venv
.venv/bin/python -m pip install site-shot
Then add this line with crontab -e:
0 6 1 * * cd "$HOME/site-shots" && . ./.env && .venv/bin/python sitemap_shots.py https://www.example.com/sitemap.xml >> run.log 2>&1
That runs at 06:00 on the first of every month, in the machine's time zone. Keep the key out of the crontab: put export SITESHOT_API_KEY=... in .env in the same folder, readable only by you (chmod 600 .env). Each run writes its images and manifest to a new shots/YYYY-MM-DD/ folder next to the script, and nothing deletes old runs, so disk use grows by one copy of the site every month. Since the sitemap is re-read on every run, new pages join the archive by themselves.
If only a few pages matter, Site-Shot's own scheduled captures need no script and no server. Each schedule is one URL, captured daily, on weekdays or weekly, with every dated image filed in your library and an optional email when the page changes past a threshold you set. Schedules come with paid plans, and each plan has a weekly ceiling on scheduled captures, shown on the pricing page. They suit the handful of pages you watch, not a whole sitemap.
Where the honest limits are
- There is no crawler and no batch endpoint. Site-Shot captures one URL per request, and the list is yours. Pages the sitemap leaves out (orphaned pages, filtered listings, anything the CMS forgot) are not captured, and a stale sitemap gives you a stale list.
- The free plan runs one capture at a time. It has one worker, which is why every loop here is sequential. Paid plans include dedicated workers, the number per plan is on the pricing page, and there a small thread pool shortens a long run.
- Pages behind a login are captured as a stranger sees them. The API renders as an anonymous visitor, so a members-only page comes back as its sign-in screen, and the manifest still says
ok. - Long and infinite-scroll pages are cut. A full-page capture stops at
max_height, 20,000 pixels at most, and content that loads only as you scroll may be missing. - Location and storage. Free-plan captures come from the default location or the US; geotargeting, storage and schedules need a paid plan. The images in this guide live on your disk, not in Site-Shot.
FAQ
How do I screenshot every page of a website?
Read the URL list from the site's sitemap.xml (its location is on the Sitemap line of robots.txt), capture each URL with a screenshot API in a loop, and record the result of every capture in a manifest. With Site-Shot, one GET request to api.site-shot.com captures one page; there is no crawler or batch endpoint, so the loop is yours.
Why does a screenshot of a missing page come back with HTTP 200?
Because the screenshot API did what it was asked: it rendered the URL, and a 404 page is still a page. Site-Shot returns an image of it with HTTP 200. Request response_type=json and read response.status_code for the status the page answered, and the error field for captures that failed outright.
How do I find a website's sitemap?
Open /robots.txt on the site and look for a line starting with Sitemap:, which points to the sitemap file. If there is none, try /sitemap.xml. A sitemap index lists further sitemaps rather than pages, because one sitemap file may hold at most 50,000 URLs.
Can I screenshot a whole website for free?
A few pages, yes: the free Site-Shot browser tool needs no signup and captures one URL at a time. For a scripted run, a confirmed account's API key comes with the free plan: 100 API captures a month, no card needed; paid plans start at $5/mo for 2,000. Each page in each layout is one capture.
Can I capture pages in parallel on the Site-Shot free plan?
No. The Site-Shot free plan runs one capture at a time, so a sitemap loop on it should run sequentially. Paid plans include dedicated workers for parallel captures, from $5/mo.
Related reading
- How to Take a Website Screenshot with Python: the SDK, return modes and a CSV-to-HTML report example.
- How to Take a Website Screenshot with Node.js: the same API from JavaScript.
- How to Automatically Screenshot a Website Every Day: scheduled captures without a script.
- Automate Website Monitoring with Screenshots: what to do with the images once you have them.
Try any page free in your browser with no signup, get an API key to run your sitemap on the free plan, or compare paid plans.