Methodology

Methodology

The goal of this research is testing the performance of specialized WordPress hosting. The two measures to determine performance are peak performance and consistency. Peak performance is measured using load testing services (LoadStorm and K6) which emulate large numbers of users visiting a website and watching how well the hosting responds under these stressful conditions. Consistency is measured using uptime monitoring to make sure that the servers remain available for a longer duration of time.

There are some non-impacting measurements taken as well that are recorded to help inform more advanced users and potentially be used to create a benchmark at a later time. Geographic response times from multiple (12) locations using WebPageTest. The WPPerformanceTester performs PHP benchmarks and WP benchmarks to see how fast the underlying server performs for different kinds of operations. SSL test checks how secure their SSL certificates are. The Internet.nl and Mozilla Observatory tests are a general website security test. These measures don’t impact the Top-Tier WordPress Hosting status we give, nor the Honorable Mention status.

Uptime monitoring runs for a minimum of three months. The primary monitors were HetrixTools and Uptime Robot. A backup monitor is run using the open source PHP Server Monitor in case there are drastically different results between the two primary monitors.

The hosting packages are as close to default as possible. In some circumstances where turning on performance enhancements is very simple, e.g. clicking an option to turn on caching, this will be done. These changes must be obvious or communicated clearly in the welcome email so that every user is guaranteed to at least be presented the option. Otherwise everything is as-is for new customers upon sign up to avoid, as much as possible, the extra benefit of companies knowing they are being tested.

The exception to this rule is the Enterprise tier ($500/month+). This exception for the Enterprise tier is because at a certain level, a higher degree of service is expected. Customers spending large sums of money are generally on-boarded and optimized by hosting companies. As such, all companies competing in the Enterprise tier are allowed to optimize the package as much as possible to maximize the performance of the site. Sites in this tier generally do well and customers looking to buy at this level should investigate the results deeply to see which company or companies best fits their needs.

This methodology isn’t perfect. One of the most common complaints companies participating have is that it doesn’t encapsulate their service well because they do ‘something special’ for every client. This may be true, but it’s hard to allow this behavior because the difference between ‘every’ client and ‘this specific test’ isn’t measurable. It also relies on good faith from the companies because load testing servers requires permission to do at a meaningful scale. These type of tests often trigger security measures and can impact existing clients. Performing unauthorized load tests would be unethical and potentially damaging towards other customers which is why we only test with permission. Working with the companies being tested is often a necessity to do these tests properly. Packages are compared to what was submitted and checking for cheating is done regularly. The comparison is designed to be as apples-to-apples as possible.

The latest testing was done on WordPress 5 running 2021 theme.

Full Dummy site (downloadable) with instructions is available at http://wordpresshostingbenchmarks.reviewsignal.com/.

Test Configurations

All tests were performed on an identical WordPress dummy website with the same plugins except in cases where hosts added extra plugins or code.

LoadStorm

The process for LoadStorm will be a scaling user test based on the pricing tier. The simulated users will hit the homepage, hit the login page, login, hit a few pages and posts. The test duration was 30 minutes and scaled from 500 to start to n,000 users over 20 minutes and sustained peak load for 10 minutes.

Price Tier # Users
<25 2,000
25-50 2,000
51-100 3,000
101-200 4,000
201-500 5,000
Enterprise 10,000

K6 (Formerly Load Impact)

K6 ran a script that requested the frontpage of the test site. It scaled from 1 to n,000 users based on the price tier over a 15 minute duration. It was designed to emulate the old Blitz.io test of simply hammering the cache.

Price Tier # Users
<25 1,000
25-50 1,000
51-100 2,000
101-200 3,000
201-500 4,000
Enterprise 5,000

Script:

import { sleep } from 'k6'
import { Rate } from 'k6/metrics'
import http from 'k6/http'

// let's collect all errors in one metric
let errorRate = new Rate('error_rate')

// See https://k6.io/docs/using-k6/options
export let options = {
  batch: 1,
  throw: true,
  stages: [
    { duration: '15m', target: 1000 },
  ],
  ext: {
    loadimpact: {
      distribution: {
        Virginia: { loadZone: 'amazon:us:ashburn', percent: 10 },
        London: { loadZone: 'amazon:gb:london', percent: 10 },
        Frankfurt: { loadZone: 'amazon:de:frankfurt', percent: 10 },
        Oregon: { loadZone: 'amazon:us:portland', percent: 10 },
        Ohio: { loadZone: 'amazon:us:columbus', percent: 10 },
        Tokyo: { loadZone: 'amazon:jp:tokyo', percent: 10 },
        Sydney: { loadZone: 'amazon:au:sydney', percent: 10 },
        Mumbai: { loadZone: 'amazon:in:mumbai', percent: 10 },
        Singapore: { loadZone: 'amazon:sg:singapore', percent: 10 },
        Brazil: { loadZone: 'amazon:br:sao paulo', percent: 10 },
      },
    },
  },
}

export default function () {
  let params = {
    headers: { 'X-CustomHeader': '1' },
  };
  let res = http.get('https://example.com', params)

  errorRate.add(res.status >= 400)

  sleep(1)
}

WebPageTest.org

Tests were run from 12 locations. 2021 test locations were Virginia, California, Salt Lake City, London, Frankfurt, Cape Town, Singapore, Mumbai, Tokyo, Sydney, Brazil, and Dubai using EC2 instances where possible.

Uptime Monitoring

Uptime was monitored for at least three months for the homepage of the site. HetrixTools and Uptime Robot were used to monitor uptime at 1 minute intervals.

WPPerformanceTester

This plugin will run its performance test. Plugin is available at WordPress.org.

Qualsys SSL Report Grade

The tool is available at https://www.ssllabs.com/ssltest/. Every single B grade this year was for the following reason according to Qualsys: “This server supports TLS 1.0 and TLS 1.1. Grade capped to B.

Internet.nl

The tool is available at https://internet.nl/test-site/. The test is designed to check if the server is up to date with latest security settings and gives a score.

Mozilla Observatory

The tool is available at https://observatory.mozilla.org/. The test is designed to give a grade based on how safe and secure the server configuration is.

Ratings

There are two levels of recognition awarded to companies that participate in the tests. There is no ‘best’ declared, it’s simply tiered, it’s hard to come up with an objective ranking system because of the complex nature of hosting. These tests also don’t take into account outside factors such as reviews, support, and features. It is simply testing performance as described in the methodology.

Top Tier WordPress Hosting (Full Star)

This is awarded to companies who maintain 99.9% uptime throughout the entire testing and show little to no performance degradation during load testing, primarily focused on error rate and consistent response times. Error rates above 0.1% and response times above 1000ms will keep a company away from achieving Top Tier marks.

Honorable Mention (Half Star)

Honorable mentions are given to companies that came close to Top Tier status but for one reason or another fell just slightly short. This is for companies struggling slightly on a load test – a low but not insignificant number of errors, slightly increased response times or spikes during the test but an otherwise solid performance.

Changelog

2021

  • Updated theme to 2021 theme.
  • Replaced Load Impact with K6 (same company, new product)
  • WPT locations switched based on availability (Salt Lake City, Cape Town, Dubai)
  • Added Internet.nl and Mozilla Observatory tests.

2020

  • Added additional testing location (Israel) on WebPageTest increasing locations from 11 to 12.
  • Self-hosted an uptime monitor as a third (backup) to compare against, if discrepancies existed between UptimeRobot and StatusCake, using PHP Server Monitor.

Costs and Price Tiers

These tests are expensive to run and I refused to accept any hosting sponsorships. There is a participation fee involved to cover the costs. Every company pays the same amount based on the price tier of the product entered into the testing.

There is a re-testing fee if the load tests require more than two attempts for whatever reason. Load testing is the primary cost involved with testing, and if a company fails twice for a legitimate reason (almost exclusively security related), then re-testing is allowed to accommodate dealing with security measures with the associated fee based on tier.

All fees paid will be documented publicly for posterity.

The table below lists all six price tiers, the testing fee associated with each tier and the re-testing fee associated with failed load tests. All plans are listed in their retail price range, no first-month/first-year/first-billing period/sale prices will be used in figuring out which tier a plan belongs in.

Pricing Tier Price Re-Testing Fee
<$25/month $100 $50
$25-50/month $100 $50
$51-100/month $150 $75
$101-200/month $200 $100
$201-500/month $250 $125
Enterprise ($500/month) $500 $250

Testing Fee Disclosures

All companies paid the same fee based on the pricing tier they competed in. The following companies were re-tested and the number of tests and associated fees are documented below.

Company Extra Tests Total ($)
20i 1(<25) 50
A2Hosting 1(25-50) 50
BlueHost 3(<25) 150
EasyWP by NameCheap 1(<25) 50
InMotion Hosting 1(<25) 50
IONOS 2(<25) 100
RAIDBOXES 1(<25) 50
RSHosting.com 1(<25) 50

 

Notes – Changes made to Hosting Plans

Almost every company had to disable security measures of some sort.

Many companies had caching turned on from either a click, welcome email instructions or other obvious way that was presented clearly to new users. Some of the instructions might be inside wp-admin, others had features in the control panel to turn on caching/performance optimizing. If it wasn’t made explicitly obvious it did not count as default.

Special note about Jelastic

I almost didn’t let Jelastic participate because their billing model makes it difficult to do apples-to-apples comparisons. However, they came up with predefined packages which fit the price tiers in this testing. All Jelastic tests were done with multiple partners: CloudJiffy, Infomaniak, SaveInCloud, and Scaleforce. They also provided the following statement:

“Jelastic is offered across a distributed network of hosting partners worldwide (https://jelastic.cloud/), thus the infrastructure performance and price for resources vary based on the chosen service provider. WordPress can be installed as packages with fixed price or using dynamic resource allocation with Jelastic standard pay-per-use approach (https://jelastic.com/pay-as-you-use-cloud-pricing/).”

 


About the Author

Kevin Ohashi

Kevin Ohashi is the geek-in-charge at Review Signal. He is passionate about making data meaningful for consumers. Kevin is based in Washington, DC.

Recommended Articles

Want updates sent to your email?

Subscribe to our Newsletter