5 Ways to SEO Optimize Your Site with Screaming Frog

SEO Optimization with Screaming Frog Crawler software
Screaming frog is a a useful SEO spider tool that can be used to crawl your website's links, images, CSS, scripts, and apps from the SEO perspective. If you didn't know about it already, then where have you been? :). It is a small desktop software that will let you crawl an entire site on demand, and will help resolve common SEO issues with your website. But crawling isn't all it's good for. It can do much, much more. In this post, we'll show you some of the most important uses of the Screaming Frog SEO Spider tool.

The functionality that Screaming Frog provides is immense. And you can mix-and-match some features to create functionality of your own - the possibilities are countless. We will attempt to create a power tutorial on Screaming Frog soon. But for the time being, just check out these 5 great ways to SEO Optimize your website with Screaming Frog.

1. Verifying crawl errors

Google Webmaster Tools can be often slow to respond to the changes on your site. For example, it might not find all the fixes you've made to your broken links, You can, however use Screaming Frog to find out broken link problems such as 3xx (eg 301), 4xx (eg 404s) and 5xx. At the top, there's a bar where you can insert the URL of a website you need to crawl. Once crawled, you can filter the results by HTML content only, and then if you go to Advanced Export, you can export the links based on their status codes (3xx, 4xx, 5xx etc).

You can, for example export your 404s, and then compare them with the ones you get in Webmaster Tools. You can also take all links from the Webmaster Tools (export as CSV, then copy all links and paste them in a text document). Now, if you go to Mode in Screaming Frog, and select List instead of Spider, you will be able to import a text file, which you can then check for errors. This will tell you all the 404s that are actually there, and will eliminate all the fixes that Google Webmaster Tools missed out on.

2. Creating a sitemap

You can use Screaming Frog to create a sitemap. All you have to do is, go to Mode tab at the top, and click on Spider. Then, from the configuration menu, click on Exclude. Here, you can exclude certain types of URLs. For example, WordPress often adds something like /?replytocom=123 when a user comments (for eg, domain.com/my-post/>replytocom=12). You don't want such URLs to be in your sitemap. So in the exclude menu, you can add this "\?repytocom." (without quotes) to exclude such URLs. Check the Screaming Frog User Guide to find out more about excluding URLs.

Now do a full crawl of your site, and once that is complete, simply export the data by clicking on Advanced Export >> XML Sitemap.

3. Checking a sitemap for errors

Similarly, you can check an XML sitemap for errors. If you have your XML file with you, go into Mode, and select List. Then, Select the XML file, and import it into Screaming Frog. Let it crawl the links. Once done, it will show you all broken links, and any other such problems. This can really help keep your sitemaps fresh and healthy.

4. Check for successful site migration

When you change your site's domain, Google Webmaster Tools often goes berserk, giving tonnes of 404 errors. You can, however, use Screaming Frog to do it quickly for you. If you have a list of your old URLs, you can import them in SF, and then find out which of them are giving you 404s. You can do a full-site crawl as well to find some of the broken links. And you can also import data from Google Webmaster Tools and then check them against any 404 errors.

5. Building links

As we discussed in our link-building exercise, you can generate authentic backlinks to your site by helping other webmasters out. If you do a site crawl for another site, and give them a list of their broken links (source and destination), you can fall into their good graces. It is indeed a great way to increase your outreach and build connections, because it tells others that you're actually reading their blog/website, and care for it. You can use this strategy to build backlinks :)

Bonus: Check out the following post, which explores yet another way to use Screaming frog for backlinking.

Did you like this post? Got any questions? I know this might be very confusing, so don't hesitate to ask in the comments section below. We'd love to help. Cheers :)

Need Quick Help within 24 Hours? ASK NOW

If you don't want to get yourself into Serious Technical Trouble while editing your Blog Template then just sit back and relax and let us do the Job for you at a fairly reasonable cost. Submit your order details by Clicking Here »

27 comments : Post Yours! Read Comment Policy ▼
We have Zero Tolerance to Spam. Chessy Comments and Comments with 'Links' will be deleted immediately upon our review.

  1. This is a very nice program which will help every bloggers in the blogosphere. I did not knew that such a desktop software exist. Keep updating us with good articles like this.

    Regards, Admin of

  2. Hi Qasim,

    I never regretted subscribing to your latest updates!!.

    One of the most essentials uses of the screaming frog tool is the ability to spy on one's competitors. With this, one can get to know the backlinks e.t.c.

    One way the tool helped me is the fact that I was able to detect five broken links ( and this broken links were not detected by the broken link checker plugin present on my blog).

    The broken links were found in my article that I posted about facebook account opening and sign up tips.

    Thanks for sharing this.

  3. I'm looking for this tool, is it free?

  4. your posts are really amazing...

    I have new blogs...

    Tamil kama kathaigal

    Crore jobs for all people

    These blogs are new blogs. Now my doubt is how to increase google seo for my blogs. can any one help me??

  5. Nice article..Thanks for sharing..

    Dixit [Triposoft.com]

  6. This is a choice, should you not possess the resources to cover a proper internet tracker the likes of Raven SEOMoz or tools. Monitor your website for a search engine and returns a report. Csv file that may be utilized as a foundation for the suggestions and reports of the webpage. The free version enables you to monitor up to 500 pages, but you can always help identify issues that may be influencing the whole website. If more is desired, the full version is just $ 99 a year.

  7. You've showed a new thing to us from the world of SEO. It is really a nice thing to Optimize SEO in a creative way. Thanks for such a great trick.

  8. Great way which provide helps for Seo experts.Link building is most important and its should be do very carefully.SEO expert in pakistan

  9. You can also use this too to find bad backlinks for your domain, for example export your list of links from Ahrefs. Load it into the Screaming frog, then scan the links. I usually remove sites which have foreign domains such as .jp and any that have a high OBL count. Overall a really nice app. IMSoldiers Internet Marketing Forum

  10. The Discount Code for Host Gator given on the top sticky floating HG50 is not a valid Code. More over I found that The Time left has suddenly dropped from 20 Hours to 5Hours Left with in a span of 1Hour..

  11. Its really Nice Information And Tools Thank you For Sharing

  12. i never heard this tool, by the way thanks for letting us know .

    Wordpress tips

  13. Plenty of things I need to know in the field of blogging and one it was answered in this great article. Thanks for the informations.



  14. Thank you for sharing such a nice post.

  15. Very Nice and amazing !! I used it for my projects
    cheap web hosting
    web site development

    Love you maaan

  16. Creating a sitemap is much more effective,Try it 1st :
    Free Back-link Generator

  17. Keep up the good work , I read few blog posts on this site and I think that your blog is really interesting and has got loads of useful information.
    Directory Submissions

  18. very helpful and unique content
    thanks for sharing
    Faltu The Community

  19. Very useful SEO tools. I think, I could easily rank with this tool. Thank you very much.

    Check out the easiest method to write seo blog post.

  20. As a SEO we developed the site with our new tools and new way of designing it. To be more friendly user and more attractive.

  21. This is the really good way for seo optimized. thepclife.com this is one of the site for pc solution. You can visit for more information.

  22. Invaluable SEARCH ENGINE OPTIMIZATION methods. I'm sure, I could truthfully quickly get ranking using this type of instrument. Thanks very much.
    SEO Back Links Service Providers | Online Marketing Solutions

    Article Submission Sites List
    Press Release Sites List
    Classified List
    Php Motion

  23. This comment has been removed by the author.

  24. This comment has been removed by the author.

  25. More and more companies are outsourcing their content strategies to professional bloggers and article writers. So how do you know if that’s an approach that makes sense for your business? the seo source