Screaming Frog: The SEO tool explained in detail

Whether large e-commerce websites, complex service provider sites or lean corporate presences: with Screaming Frog, you can examine your website down to the smallest detail. And so should you, because only if you know your site's technical SEO problems can you specifically fix them. In this guide, you will therefore learn everything you need to know about the tool and how you can use it to improve your SEO performance.

The basics of the SEO spider tool: What is Screaming Frog?

Screaming Frog is a so-called website crawler. That means it simulates the Googlebot and scans all accessible pages on your website. In doing so, the tool collects technical and content information, such as meta data, status codes, internal and external links, canonicals, redirects, or even the structure of your site. By the way, the official name of the tool is Screaming Frog SEO Spider. That's because the Spider Tool “crawls” through your site like a spider and pulls out all SEO-relevant information. You can crawl individual URLs, exclude specific pages, or even search specifically for incorrect meta descriptions. There is rarely so much power in a single tool, especially not in the free version, which already does quite a bit with Screaming Frog.

Who is Screaming Frog particularly suitable for?

Screaming Frog is not just for SEO experts, the tool also offers marketing teams, web developers and e-commerce managers enormous added value. This is particularly true for large websites with hundreds or even thousands of URLs. Here you don't need a pretty dashboard, but reliable, clear data. Online shops with many product variants are a good example of this. This can quickly lead to duplicate content or unnecessarily crawled pages. However, Screaming Frog shows you exactly where this is happening. With targeted measures such as canonical tags or crawl blockers, you can save your crawl budget and make your website scalable in the long term.

What makes Screaming Frog different from other SEO tools such as Semrush, Ahrefs & Co?

When it comes to detailed insights into the technical structure of your website, there is hardly a comparable alternative. Other tools like SEMrush or Ahrefs may provide you with nice dashboards and visual reports, but they often just scratch the surface. Screaming Frog, on the other hand, goes deep. It analyses exactly where errors occur and gives you the opportunity to fix them directly. Many SEO professionals therefore use Screaming Frog every day in their work to carry out audits, check websites for SEO errors or to accompany redesigns. And the best part: You can export all data and process it further in Excel or Google Sheets. For anyone who likes working with real numbers, this tool is therefore ideal.

The benefits of Screaming Frog for technical SEO

Why should Screaming Frog be an integral part of your SEO strategy? It's simple: It is fast, precise, individually configurable and provides exactly the data you need. No unnecessary bells and whistles, no data distortion due to external interpretations. You get raw data. And that's exactly what you need if you want to do high-level technical SEO. The most notable advantages also include:

• Complete crawl control: You decide what gets crawled and what isn't.
• Level of detail of data: Status codes, redirect chains, meta data, canonicals... it's all there!
• Custom Extraction: Draw individual information from specific page sections.
• Export functionalities: Continue working in Excel, Data Studio, or custom tools.
• Easy integration, for example with Google Analytics, Search Console & PageSpeed Insights
• Flexible spider configuration for large websites

Getting started with Screaming Frog SEO Spider

Getting started is surprisingly easy. You download the software directly from the official website, install it just like any other desktop tool, and you're ready to go. For smaller projects, the free version is often enough. On the other hand, you are more likely to need the paid license for larger projects or if you want to use features such as the JavaScript crawl or the API connection directly. After installation, you should immediately take a look at the settings. Here you can configure Screaming Frog specifically for your projects. Do you only want to crawl certain pages? Then create a list of URLs. Do you want to ignore CSS or JavaScript? That is also possible. Using the “Spider Configuration” menu, you can set what is crawled and how deep the crawl can go.

Screaming Frog: The interface simply explained

At first glance, Screaming Frog looks like a classic spreadsheet tool. You'll see navigation at the top, a filter bar below, and then a big list of URLs. Each line represents a URL and each column contains information, such as the status code, the page title, the meta description or the canonical link. Using the tabs above, you can then specifically analyze specific URL types or problems: HTML, images, redirects, duplicate content, internal linking, and much more. In the right sidebar, however, you will receive additional information about the selected URL, depending on your selection. It is also particularly practical that you can see directly which pages link to a specific URL and whether this link may be incorrect.

Optimally configure Screaming Frog Crawl

The true power of Screaming Frog unfolds when you set the crawler to suit your needs; after all, not every page on your website is equally important. From the “Configuration > Spider” menu, you can determine whether, for example:

• crawled noindex pages,
• analysed external links,
• Canonicals observes and
• AJAX & JavaScript content is loaded.

These settings significantly influence what data you get and how you have to interpret it. For shops with many filters, for example, you should absolutely rule out crawling parameter URLs that lead to the same content anyway.

Which pages should you crawl and which not?

Not every page has to be included in every crawl. Especially for large websites with over 100,000 pages, it is worthwhile to regularly analyze only certain areas, such as category pages, product pages or important information pages. You can also specifically control the crawling of specific directories or subdomains via the configuration. Also use Custom Extraction to specifically analyze pages with specific HTML classes or IDs, e.g. for pages with schema markup or conversion elements. This gives you exactly the information that is important for your optimization.

Carry out the first crawls: It's easy

Do you want to get started? Then enter the URL of your website in the input field at the top and click on “Start.” The Screaming Frog is already beginning to crawl the URLs. Depending on size and server speed, this takes a few minutes to several hours. Especially with large websites, you should therefore also carry out a limited crawl. For example, limit the depth or set targeted filters so as not to consume the entire crawl budget right away. As soon as the crawl is complete, you can analyze and export the data and search specifically for errors or optimization options.

Performance optimization: Dealing with large websites

Large websites have special requirements when it comes to crawling. That's why you should definitely:

• Adjust the crawl limit
• Set Memory Mode to “Database Storage”
• Only share relevant directories and URLs
• Select a user agent sensibly (for example, Googlebot)
• Reduce crawl speed so as not to overload servers
• Optional professional tip: Divide the website into thematic clusters and crawl them separately. This saves time and at the same time provides you with more targeted data for individual areas.

Screaming Frog: An SEO tool with real depth

In the following, we'll take a closer look at where the Screaming Frog tool particularly shines, and how you can use it to check meta data, analyze the internal link structure, uncover redirects and 404 errors, check external links, create sitemaps, and even perform a keyword check.

1. Analyze meta data with Screaming Frog: Keep track of titles, descriptions, and more

One of the first tasks of an SEO analysis is to check the meta data. Screaming Frog shows you at a glance which title tags are too long, too short or duplicate. The same goes for meta descriptions. Especially when it comes to extensive websites, many quickly lose track of things. This is where the tool helps to systematically identify errors. With a targeted crawl, you can export a complete list of all meta data on your website in just a few minutes. This data is easy to filter, analyze, and prioritize. It is best to use the “Bulk Export” function to optimize the meta data and transfer it straight to the CMS. This is how you close gaps in SEO performance sustainably.

2. Analyze and optimize internal linking with Screaming Frog

Screaming Frog shows you exactly how many internal links point to a specific page, which anchor texts are used and whether the structure is logically structured. This ensures that the link juice is distributed efficiently and that Google finds all important pages on your website. Once you've used the tool to identify the pages that have few or few clicked internal links, you can take targeted measures to improve the links, for example through link boxes, navigation extensions, or thematic text links. Another advantage that Screaming Frog helps you recognize whether orphaned pages exist, i.e. pages without incoming links. These “orphans” are critical from an SEO perspective as they are difficult to crawl and index. It is therefore your task here to integrate or deliberately exclude these pages.

3. Detect redirects, 404 errors & status codes

No SEO analysis is complete without verifying redirects and status codes. Screaming Frog therefore automatically identifies all 301 and 302 redirects for you, shows you redirect chains and detects 404 errors and 5xx error pages. This is particularly useful for website relaunches or structural changes, as it can quickly happen that old URLs are redirected or removed. Without a clean redirect structure, you lose valuable traffic and also risk ranking losses. With Screaming Frog, however, you can identify problems like these early on. It is best to also use the “Response Codes” function to get an overview of all status codes. In this way, you can specifically revise problematic URLs, avoid redirect loops and ensure the crawlability of your website in the long term.

4. Identify, analyze & evaluate external links

External links are not only important for SEO, but also for your users. After all, there is hardly anything more annoying than a link that leads into space (broken link) when you were just looking for something specific or an answer. Screaming Frog therefore shows you at a glance which external links lead to 404 pages. These should be corrected or removed as a matter of urgency. You can find this information in the “External” tab, where you can see all outbound links on your site, including the status code. It's worth checking out older websites in particular, because many external sites change their structure or disappear altogether.

Relevance and quality of external links

In addition to the technical review, you should also analyze the content relevance of your external links. Screaming Frog can capture these links for you and prepare them so that you can export them to a comprehensive report. This gives you an easy way to evaluate whether your site links thematically appropriate and trustworthy sources.

5. Create & check XML sitemaps with Screaming Frog

Your XML sitemap helps search engines find your most important pages efficiently and crawl them in a targeted manner. With Screaming Frog, you have the practical feature that you can create your own sitemap directly in the tool, based on the URLs actually crawled. To do this, go to the “Sitemaps” menu and select “Create XML Sitemap.” You can now set exactly which URLs should be included. For example, only pages with a 200 status code, only indexable pages, or only pages that are linked in internal links. In particular, make sure that only URLs are included that have real relevance and should be active in the index. Pages with a “noindex” tag or redirects should be explicitly excluded. This is how you give Google clear signals and save the crawl budget.

What makes a good site map and why it's important

Especially for large websites or dynamic platforms, the sitemap is a valuable tool for quickly adding new content to the index. However, for this to work smoothly, it should be structured, contain the most important pages of your project, be updated regularly and be free of errors. With Screaming Frog, you can also check the existing site map for errors. To do this, load the sitemap file via “Mode > List” and analyze it for status codes, canonicals and indexability. In this way, you can be sure that all linked pages are easily accessible and crawlable. An often underestimated but extremely important factor for your SEO performance.

6. Use Screaming Frog for keyword research & keyword check

Screaming Frog is not a classic tool for keyword research, but you can use it specifically to identify keyword potential and review existing content. For example, you can use the custom search function to analyze whether certain keywords appear in title tags, meta descriptions, or H1 headings. Another trick: export all page titles and analyze them in Excel or Google Sheets for keyword density, relevance, or gaps. This allows you to quickly find out which pages do not yet have clear focus keywords, which is often a reason for weak rankings. In combination with classic tools such as Google Keyword Planner or Ahrefs, you can build a precise on-page strategy. They let you know exactly which keywords you're targeting. You can then use Screaming Frog to check whether they are actually properly integrated. By the way, this also applies after every content update. The tool allows you to see whether the new keywords have been placed correctly and whether there are no technical barriers such as “Noindex” or faulty canonicals in the way.

Linking Screaming Frog with external SEO tools: Does that make sense?

Screaming Frog offers the option to connect to Google Search Console and Google Analytics. This gives you real performance data from your pages directly into the tool and allows you to analyze exactly which pages bring a particularly large amount of traffic and whether they also rank for the right keywords. By combining crawl data and performance indicators, you get a holistic picture of your website's SEO performance. This is particularly helpful if you want to develop content in a targeted manner or want to find out why certain pages are stagnating in the ranking.

Screaming Frog: API & connections to other tools

With the API integration in Screaming Frog, you can integrate data from Google Analytics, Search Console, PageSpeed Insights, or Ahrefs directly into your crawl analysis. This means that you not only see technical errors, but also get performance indicators such as sessions, bounce rates, or clicks, right at the URL level. For example, you find a page that has lots of internal links but no traffic. With API integration, you can immediately see whether this page has bad content or whether there are technical reasons. This helps you to optimize the right pages and content in a targeted manner.

Automated audits & reporting with API support

The API is often particularly interesting for SEO agencies and larger projects. This is mainly due to the fact that you are like this:

• carry out regular audits,
• generate automated bug reports,
• Fill custom dashboards and
• be able to document historical developments

In combination with tools such as Data Studio or Looker, you can build individual reports that show you the status of your website at any time.

Practical example: Technical SEO audit with Screaming Frog in the online shop

Screaming Frog shows its full strength particularly when it comes to large, complex websites, such as online shops with thousands of sub-pages, many filters, product variants and dynamic content. So let's imagine that you manage an online shop with over 20,000 product pages. The task: Find technical errors, optimize internal links, improve meta data and ensure faster indexing.

Step 0: Your step-by-step crawl guide

Before you get started, you first need to ensure a clean crawl configuration. After all, you don't want to waste resources, especially with large shops, by having irrelevant URLs, session parameters, or duplicate filters crawled.

Optimize crawl setup: Here's how

• In the “Configuration > Spider” menu, you first deactivate unnecessary resources such as PDFs or external links.
• In the “Include/Exclude” section, you narrow down the crawl to relevant directories (e.g. /products/, /categories/) to ensure focus and performance.
• Optionally, you can use the Custom Extraction area, for example, to directly read out product prices or structured data.

The result: A focused crawl that analyses the most important SEO-relevant page types in just a few hours instead of days.

Step 1: Systematically identify incorrect metadata

After the crawl, it's time to analyze. In the “Page Titles” and “Meta Descriptions” tab, you can find at a glance:

• Missing metadata
• Duplicate or too long title tags
• Descriptions under 70 or over 155 characters

Especially in shops, duplicate metadata is often generated by similar product pages. As an SEO specialist, you should therefore better export, clean and, if necessary, automate this data with individual templates (e.g. “Product Name | Category | Shop Name”). Metadata optimization definitely belongs high up in a well-structured audit table of contents, as it can be implemented quickly and has great leverage on the click rate (CTR).

Step 2: Analyze status codes and redirects

In the “Response Codes” tab, you filter specifically:

• All 404 errorsto identify orphaned or outdated pages
• 5xx errorthat indicate server issues
• 301 redirectswhich should be tested on chains or loops

It is also best to export a redirect chain via “Reports > Redirect Chains”. There, the tool shows you which pages are routed through several stations. However, this is bad for load time, user experience, and crawlability.

Step 3: Optimize internal linking

In the “Site Structure” module or in the “Inlinks” tab, you can immediately see:

• Pages with less than 2 internal links
• unimportant pages linked excessively often (e.g. terms and conditions, legal notice)
• incorrect or non-speaking anchor texts

The goal should be that important categories and products are sufficiently internally linked. Screaming Frog also allows you to export link targets and anchor texts in order to revise them in a targeted manner.

Step 4: Check dynamic content & JavaScript issues

Especially in shops, product information is often hidden in pop-up tabs or tabs, often reloaded via JavaScript. However, with the JS rendering feature enabled in the licensed version of Screaming Frog, you can:

• Crawl content in tabs, filters, or modal windows
• check whether this content is indexable
• Make hidden content visible and optimize accordingly

The result is often surprising: Many pages lose SEO potential because Google is unable to read this content completely.

Step 5: Check keyword relevance and placement

With a prepared keyword set, you can now use Custom Search to specifically check:

• Whether there are focus keywords in the title, in H1, meta description or in the body
• Whether product pages contain relevant terms or only use manufacturer texts

For SEO specialists in particular, this is a valuable function for specifically aligning content with search intentions and increasing semantic relevance.

Step 6: Create & compare a sitemap

After the final crawl, you then use Screaming Frog to create your own XML sitemap, which:

• contains only indexable pages
• excludes all redirects
• can be compared with the sitemap submitted in the Search Console

Here you can also segment sitemaps by page type (e.g. products, categories, blog), which helps you analyze them even more specifically and optimize them faster.

Conclusion: That's why Screaming Frog is worthwhile for your website

Screaming Frog gives you the opportunity to collect, analyze and systematically optimize all important SEO data of your website in real time. But that's not all. The tool also scores points in terms of depth of analysis and flexibility. With it, you can easily analyze small blogs as well as large portals with hundreds of thousands of URLs. Thanks to the numerous configuration options, from crawl depth to spider settings to targeted custom search, the tool adapts perfectly to your individual requirements. However, one of the biggest highlights remains the ability to crawl modern web technologies such as CSS JavaScript. This is particularly advantageous for dynamic pages that only load content later via JavaScript. You can see whether Google is really seeing what your users see and can react accordingly. And that is the basis for good rankings, increasing traffic and stable SEO performance. So if you're looking for a tool that really helps you, regardless of marketing phrases and dashboard blinders, Screaming Frog is just the thing for you.

If you need assistance with the application or would like to have a professional SEO audit carried out for your site, as an SEO agency, we are also happy to assist you with our experience, know-how and tailor-made solutions.

FAQ: The most common questions about Screaming Frog

Below you'll find answers to the most common questions about Screaming Frog.

How much does Screaming Frog cost and is the free version sufficient?

The free version is usually sufficient for smaller websites up to around 500 URLs. For larger projects or advanced features such as JavaScript crawling and API integrations, however, we recommend the paid license, which is currently 245 euros per year.

How often should you crawl with Screaming Frog?

Depending on the size and frequency of changes on your website, we recommend a crawl at least once a month. For ongoing optimizations or relaunch projects, a weekly crawl can also be useful.

Can I also analyze images with Screaming Frog?

Yes, the tool shows you which images are used on your site, whether ALT texts are missing and how large the files are, all important factors for loading time and accessibility.

Is Screaming Frog GDPR compliant?

Since Screaming Frog runs locally on your computer and does not transfer any data to external servers, its use is generally GDPR-compliant. However, you should make sure that you do not store or process any personal data.

What alternatives are there to Screaming Frog?

Good alternatives include Sitebulb, JetOctopus or OnCrawl. All three offer similar features but are usually more expensive or complex. Screaming Frog therefore remains our tool of choice for in-depth technical SEO checks.

‍