Managing Broken Links and 404s in WordPress

Broken links create a horrible user experience. Who wants to click on a link, only to find that it goes nowhere? That the awesome article you were all set to read doesn’t actually exist?

Unfortunately, the natural decay of links (also often referred to as link rot) occurs all too often. Link rot happens for any number of reasons: domains expire, websites are abandoned, incorrect URLs are used, and websites are restructured using new URLs. This happens to both outbound and inbound links. And, even though best practice dictates the use of 301 redirects in the case of website restructures, not everyone sets them up. Or, if they do – they’re often set up incorrectly.

This shouldn’t stop you from linking to other sites. Linking to other websites is an important component of what the web is all about. Some people tend to think that linking out to third party websites will cause visitors to leave their site, reducing their visitor stats, but linking out is exactly what the web was designed for. You shouldn’t be scared of it, rather, you should embrace the value that enriching your content offers. Providing your visitors with related content or a deeper dive on a given topic is a far better user experience than dumping users at a dead end. This is just my opinion, I know there are different schools of thought when it comes to the topic of outbound links.

So, if you can’t control other people’s websites, and you shouldn’t stop linking out to other websites, then what can you do to make sure you’re not sending people to non-existent pages? Well, it’s pretty simple really: you can control your own site by performing regular link maintenance and keeping your outbound links in check.

The management of broken links is an integral part of good WordPress maintenance. And, thanks to a number of plugins and tools at our disposal, it’s becoming increasingly easy to automate the process of link maintenance these days. So, without further ado, let’s take a look at some of these tools.

Link Checking WordPress Plugins

Broken Link Checker

As the name suggests, this plugin has been designed to check all the content on your WordPress website (from your posts and pages, as well as comments) for broken links and missing images. When you first install this plugin on your site, it will begin by checking all of your links. Depending on the size of your site this can take a while. This plugin has lots of configuration options, you can set up automatic notifications either via your WordPress dashboard, or by email, and can even set it to display broken links differently. It can also prevent search engines from following any broken links that might exist on your website, which is handy when you have a lot of user generated content.

Once you have this plugin installed and configured, you can manage any broken links from within the plugin interface. The ‘Broken Links’ tab in your WordPress dashboard will contain a list of all problem links. Each link will then have four different options available:

• Edit URL: which lets you change the URL of the broken link, across all instances included throughout your website. This is quite a handy option for correcting issues like typos or small mistakes (like using .com or .com.au).
• Unlink: this allows you to leave the URL on your website page, but to remove the hyperlink (so your website users can’t actually click on the link).
• Not Broken: this allows you to manually change the classification of a link. It is quite helpful if you know that a link isn’t actually broken. For instance, if a website is temporary down for maintenance or there is a glitch in the system.
• Dismiss: this removes the link from the ‘Broken Links’ tab, and checks it as normal. It can be used if you want to confirm that a link is broken, but not do anything about it for the time being.

With over 400,000 active downloads (and more than 3.2 million in total), and a 4.1 star rating, this is the most popular broken link check plugin on the market.

A quick word of warning on this plugin though, according to some users it can cause high load on the server side of things. In fact, WP Engine has it on their list of disallowed plugins because it requires an high number of HTTP requests. Make sure you’re monitoring your server resources if you check this one out.

LinkPatrol

Next up on the list, we have LinkPatrol, a plugin that cleans up your outgoing link profile. Created by the team over at Search Engine Journal, LinkPatrol works on the basis of a simple, three-step process:

1. Scan: the plugin will scan all the content on your website. This may take a few seconds to complete, but once it’s done, you won’t have to do it again. The plugin will automatically scan any new posts and pages as they are created.
2. Review: the plugin then gives a report on all your links, enabling you to sort by the domain, the author, and even the anchor text that you have used. You can even export your report as a CSV file.
3. Take Action: based on the information included in your comprehensive report, you can then determine what it is that you need to do quickly and easily. For instance, you can strip or nofollow all links from particular domains. The options really are quite immense.

LinkPatrol is not a free option; there are three different price points:

• Blogger: $50 one-off fee, which allows you to use the plugin on one site, and comes with one year of support and updates
• Consultant: $100 one-off fee, which allows you to use the plugin on five sites, and comes with one year of support and updates
• Agency: $200 one-off fee, which allows you to use the plugin on 20 sites, and comes with one year of support and updates

Other Link Checking Tools

While the two plugins above are great options when it comes to checking all the links on your WordPress website, if you don’t want the overhead of installing additional plugins, there are plenty of standalone options. So, here are a few link checking tools, that you can use on any site.

Google Search Console (formerly Google Webmaster Tools)

A good tool for checking the broken links on your website is one that’s provided by Google itself: Google Search Console. Google Search Console is the new name for what you may already know as Google Webmaster Tools. Google Search Console is quite easy to use, all you need to do is log into your Google account and go here. On the homepage, you’ll be greeted with the dashboard for your website or websites. Underneath the ‘Current Status’ heading, you’ll see a section called ‘Crawl Errors’. If you have any broken links that Google detects when it crawls your website, they’ll be displayed here, making it quick and easy to pinpoint (and then fix) most of your broken links.

Screaming Frog SEO Spider

Screaming Frog SEO Spider (commonly just referred to as Screaming Frog) is a popular desktop spider/web crawler. Since it’s a desktop application, you’ll need to download the program directly onto your computer. It will then crawl all aspects of your website. It will then analyse it from both a user and an SEO perspective (as its name suggests). It enables you to quickly and easily crawl, analyse and audit the content of your website, all from an SEO perspective. It presents detailed reports, which you can slice and dice in any manner that you like. And of course, it includes extensive details on broken links.

There are different versions available of Screaming Frog, available for Windows, OS X and Linux:

• Standard/Lite: which is completely free to download and use. However, it does have some limitations. It can only crawl 500 URLs in any one project, and you don’t have full access to the configuration options. For instance, you can’t save your crawls.
• Individual Licence: for the cost of £99 per year, you can up the ante, gaining access to all aspects of the configuration and getting rid of the 500 URL crawl limit. As a long time customer, it’s a great tool, one that I highly recommend.

Screaming Frog is used regularly as a broken link checker, offering detailed user instructions on how to use it for this purpose alone.

There are also a few competitors to the popular Screaming Frog, a few of them web-based applications, all with their own pros and cons. The web apps do cost a little bit more, but they’re worth it if you need some grunt to tackle larger projects.

• WildShark SEO Spider (currently Windows only)
• Botify (web app)
• DeepCrawl (web app)

There are also several desktop tools that specialize in just link checking, as opposed to the multi-purpose SEO crawlers listed above. I’m sure there are more, but these seem to be the most popular ones I’ve come across by far:

• Xenu’s Link Sleuth (Windows only)
• Integrity (OS X only)

LinkChecker

LinkChecker is open source project hosted on GitHub that deserves a mention. It’s a free website validator that supports recursive site crawling, cookies, restriction of link checking and regular expression filters for URLs. Given it’s technical nature, if you are a WordPress beginner, then this might not be the right tool for you. For advanced users or developers, it’s definitely worth checking out.

404s – The Last Resort

With all these tools on the market, many of them free to download and use, there really are no excuses for broken links on your website. But, as I mentioned earlier, you can’t stop what people type into the address bar, or what people use as links to your site. Plus, we all make mistakes, right? So knowing this, at least ensure that you have a brilliant 404 error page set up on your website. A ‘404’ is the HTTP response code received when a page isn’t found, something that all too often is left as a bland, server default, error message. A brilliant 404 landing page will help garner a little bit of forgiveness from your website users (or at least your users with a sense of humour!).

In your WordPress theme folder, you’ll probably find a 404.php template that you can edit, but you’ll need to check your theme settings and documentation. You can find out more information on how WordPress 404 pages work over at the WordPress Codex. You can also modify your web server configuration, such as your .htaccess to do something funky with 404s.

If you’re looking for something with a few more options, here are a few useful plugins for monitoring and managing your 404s:

• 404page
• 404 Notifier
• 404 to 301
• 404 to Start
• Redirection

If you’re suppressing errors, take extra care that everything is working as it should. Redirecting 404s can be a double-edged sword, at the very least, make sure your logging errors so you know when something goes wrong.

I’ve included below a few examples of amusing 404 pages to give you a few ideas (you can also find more here and here). As always, if you know of any other useful tools or tips related to this topic, feel free to include your comments and suggestions below.

---

---

---

---

---

---

Frequently Asked Questions about Managing Broken Links and 404s in WordPress