Server Error (5xx): what is it and how to find it?

What is a 5xx Server Error?

A server returns a 5xx server error when it can't successfully complete a client's request.

5xx means any HTTP status code that starts with 5, so among others, this covers the 500, 502, and 503 errors.

A server error (5xx) is often referred to as a “5xx server error.” These mean one and the same thing, and we’ll use them interchangeably throughout this article.

Why you should care about 5xx server errors

5xx server errors present a big issue, because they directly harm your visitor’s experience. Imagine having an eCommerce store, and your visitors keep receiving these server errors. Do you think they’ll keep trying and eventually purchase something?

No, they’ll go next door and buy from the competition.

And what about search engines like Google? If they receive 5xx server errors just occasionally, then you probably don’t have much to fear. But if the 5xx server errors are persistent, then it’s likely you’ll see a steady decline in organic traffic.

In this article we’ll cover why 5xx server errors are bad for SEO, what the most common 5xx server errors are, how to find out if your site’s server returns them, what causes this, and how to fix them.

Why every server error (5xx) hurts your SEO performance

We already know that 5xx server errors make for a very poor user experience. But they’re also bad news for your SEO performance.

Here’s why:

1. When search engine crawlers (“crawlers” from here on out) consistently receive 5xx server errors while crawling your site, they’ll want to avoid adding to the performance issues your site already has, so they’ll dial back their crawl efforts.Essentially this means that they’re lowering your site’s crawl budget. This results in your content getting (re)indexed more slowly.
   
   Don’t forget: if your content isn’t indexed, it doesn’t rank.
2. When crawlers keep receiving these 5xx server errors for your URLs, why would they grant your troubled pages high search results?Any visitors they send your way will likely have a poor user experience. So, when the 5xx server errors persist, Google will start “demoting” your URLs – assigning lower positions or even completely removing them from its search engine result pages (SERPs).
3. When pages aren’t available to crawlers, they won’t be able to (re)index them and rank them. They won’t know what the content is about, and these pages won’t surface in Google’s search results.

The bottom line: you could have the best content in the world, and the strongest links to your site, but you’ll have a really hard time ranking if your site is consistently suffering from 5xx server errors.

The most common 5xx server errors

The most common 5xx server errors are:

• 500 – Internal Server Error: the server had issues processing the request; this could have any one of a thousand reasons, often related to technical issues such as bugs or software incompatibilities.
• 502 – Bad Gateway: the request cannot be fulfilled successfully, because an upstream service (a service you’re expecting a response from) did not respond correctly. The upstream service can be running on the same machine or another machine, and it typically consists of a CGI component such as PHP and an application component such as a WordPress CMS.
• 503 – Service Unavailable: the server is temporarily unavailable, and will be available again later. Often the 503 server error is used when performing maintenance or when the server is too busy.
• 504 – Gateway Timeout: this status code is closely related to the 502 server error. In the 504 server error, the server – while operating as a gateway or proxy – didn’t receive a response from another server in time. As we mentioned above, keep in mind that your web server is just a gateway too. What if your back-end takes too long to respond to your web server?
• 509 – Bandwidth Limit Exceeded: your website has exceeded the server’s assigned bandwidth limits and thus is no longer available to the public. This frequently happens on shared hosting environments when they experience a sudden influx of visitors and/or crawlers. While this server error is frequently used, it’s not specified in any RFCs (see for yourself ).
• 524 – A timeout occurred: the Cloudflare CDN invented this server error and shows it when the origin server – your site – doesn’t respond in time. If the CDN doesn’t get a response, they can’t provide visitors and crawlers with one either. This results in the following error screen, which you may be familiar with:

How to find 5xx server errors

It’s clear 5xx server errors are bad for business, but how do you find them? We’ll kick off with the easiest and fastest one, followed by two more: using reports in Google Search Console (GSC) and log file analysis.

24/7 cloud-based monitoring

If your website is vital to your business, you need to monitor it nonstop for issues such as these 5xx server errors. You want to be alerted in case of trouble. That’s exactly what the ContentKing platform does. Just plug in your site, the monitoring starts, and you’ll be alerted through Slack and/or email when something goes down:

Find 5xx server errors in GSC’s Index Coverage report

Google Search Console’s Index Coverage report is very useful for quickly tracking down any server errors Google has received.

1. Log on to Google Search Console .
2. At the top left, choose the right property.
3. Then on the left, click on Coverage under Index.
4. Look under the Error issue category to see what URLs Google encountered 5xx server errors for:

5. Click on Server error (5xx) to jump to the overview of affected URLs.

Find 5xx server errors in GSC’s Crawl Stats report

Google Search Console’s Crawl Stats report shows you statistics about Google’s crawling history on your website. It lists what file types are requested most, what purpose their crawl request had (refresh content, or discover new content), and – this is what we’re interested in most – what “crawl response” they received.

1. Log on to Google Search Console
2. At the top left, choose the right property.
3. Then scroll all the way down, and click on ⚙️ Settings in the left sidebar:

4. On the Settings screen, click Open report:

5. Look under Crawl requests breakdown > By response to find Server error (5XX):

6. Click on Server error (5XX) to jump to the overview of affected URLs. Interestingly, it’s very possible that this report will show different URLs than the Index Coverage report does.

Find 5xx server errors in your server log files

And last but not least, you can go straight to the source: a log of all the requests that were made and what the server’s response was – your server’s log files.

While it can be notoriously difficult to get your hands on server log files, they are very valuable, because they provide a full picture of all the 5xx server errors your server has returned.

At the end of the day, the Index Coverage and Crawl Stats reports are a digest, and they are limited to the 5xx server errors Google received.

What causes 5xx server errors?

There are thousands of reasons why your server might return a 5xx server error; 5xx server errors vary widely within their class and – while there are RFC standards and the like – it’s up to developers’ discretion to choose which 5xx server error is most applicable when a certain error occurs.

Additionally, it’s important to understand these server errors can occur on different levels, such as:

1. CDN (e.g. Cloudflare or Akamai)
2. Web server (e.g. Apache or nginx)
3. Programming language (e.g. PHP or .NET)
4. Application (e.g. WordPress or Magento)
5. Plugin (e.g. a page speed optimization or contact form plugin)

Below we’ll cover some common examples of what can cause 5xx server errors, but this is by no means an exhaustive list.

Code bugs and updates gone wrong

The most common reason for 500 – Internal Server Error to occur is because of code bugs. The client makes a request, and while the server is processing that request, a bug causes some issue, and the server can’t successfully meet the request as a result.

Another common reason for the 500 server error is when running updates has gone wrong. For instance, when a new version of your CMS comes out, and you upgrade it, but some of your plugins aren’t compatible.

So provide your developers with leads about where to start debugging. Explain in as much detail as possible what went wrong, and where it went wrong.

The back-end has died

Your server will probably respond with a 502 – Bad Gateway or 504 – Gateway Timeout error if its back-end (such as the CGI component that runs the PHP powering your website) has crashed. Your server relies on its back-end to serve a page – and when the back-end doesn’t respond, it can’t produce anything.

An underpowered or misconfigured hosting platform

When you’ve got an underpowered or misconfigured hosting platform, and your server can’t handle all of the requests anymore, it’s likely to return the 503 – Service Unavailable response.

You could be a victim of your own success because your PR campaign has drawn a pile of visitors to your site. Or maybe crawlers have gone rogue, continuously hammering your site.

There’s no “one size fits all” solution to this issue. It’s best to discuss this with your developers and hosting provider. The right solution depends on your specific situation, and oftentimes includes a combination of sub-solutions. To name a few:

1. Upgrade your hosting platform (beef up your server’s specifications).
2. Enable caching.
3. Start using a Content Delivery Network (CDN).
4. Improve code performance.

And in the case of bad crawlers, you could implement a bot management solution and adjust your robots.txt to throttle bad crawlers (crawl-delay directive) or keep them out entirely (disallow directive).

Note that a crawler situation may eventually lead to the 509 – Bandwidth Limit Exceeded server error.

Tackle 5xx server errors head on

You’re bound to encounter 5xx server errors at some point. What matters most is that you quickly find out about them, and that you deal with them swiftly. Don’t let 5xx server errors hurt your SEO performance.

Monitor your website around the clock and leverage Google Search Console’s tools and your server log files to investigate and debug these issues. Keep an open mind as to the reason why you’re getting these 5xx server errors. There are literally thousands of possible reasons out there, and they can happen on various levels in your architecture.