Log file analysis for SEO: what server logs tell you
What server logs show about how search engines really crawl your site, how to verify genuine Googlebot visits, and which questions logs answer best.
· 2 min read
Most SEO tools show what could be crawled. Server logs show what actually was: every request search engine crawlers made, when, and what the server returned. For larger sites, logs answer questions no other data can.
What is in a log file
Each line records one request, usually with:
- the IP address of the visitor or crawler;
- the date and time;
- the URL requested and the method (GET, POST);
- the status code returned (200, 301, 404, 500...);
- the user agent, which names the browser or crawler;
- often the response size and time taken.
Make sure it really is Googlebot
Anyone can claim to be Googlebot in the user agent. Google's documented way to verify a crawler is a reverse DNS lookup on the IP address (the name should end in googlebot.com or google.com), followed by a forward DNS lookup on that name to confirm it returns the same IP. Google also publishes the IP ranges of its crawlers. Verify before drawing conclusions, or a scraper can distort your analysis.
Questions logs answer well
- Are important pages crawled, and how often? Pages that matter but are rarely crawled may need more internal links or a place in the sitemap.
- Where is crawling wasted? Parameter URLs, filters, old redirects and error pages that take a large share of requests. See Crawl budget: when it matters.
- What errors do crawlers hit? 404s and 5xx errors that users rarely report.
- How fast does the server respond to crawlers? Slow responses reduce how much search engines crawl.
- Did a change work? After a migration or a robots.txt change, logs show quickly whether crawling followed.
Getting started
- Get access to the raw logs from your host or CDN. On some hosting, logs are rotated after a few days, so collect them regularly.
- Filter to verified search engine crawlers.
- Group URLs by template or section (products, categories, articles, parameters) rather than looking at single URLs.
- Compare with a crawl of the site and with your sitemap: pages in the sitemap that crawlers never request, and URLs crawlers request that are not in your crawl, are both worth investigating.
Spreadsheets work for small sites; dedicated log analysers and crawlers with log integration, such as Screaming Frog, handle larger volumes. The Crawl Stats report in Google Search Console gives a summary view from Google's side.
Questions about a large site? Ask in the Technical SEO community.