Core Web Vitals: One Test Just Isn’t Enough

“Well it works on my machine.” – The age-old catchphrase that developers used when the tester found an intermittent defect, today it’s often used when people run a Google Lighthouse test on their machine and it ends up showing wildly different results to what is being reported by the team.

Google has made huge leaps in helping to make websites faster and improving website performance, and now with Core Web Vitals, they are laying down the gauntlet to businesses to ensure they take web performance seriously. 

One of the reasons for this more aggressive stance on website performance is that Google’s search crawlers take up a lot of energy and power, so every time they get us to improve our websites, they save money, it makes absolute sense for Google.

The internet is also one of the largest consumers of energy in the world, so the good of the wider world is also important in website performance, Google knows this. Reports of the amount of energy the internet uses are wildly varied because often we only count device usage and not the hops. If you are serving images from across the world in an international data centre, that’s a whole lot of electricity, and a whole lot of movement, and even if it is ‘fast’ it might not be ‘sustainable and fast’.

Internet energy consumption hops
Figure 1: Internet energy consumption hops

However, even if we can’t agree on the full impact, we can all agree that, if a website is slow and a device has to do more work to load it, it uses more energy, so fast websites make the world a better place for all sorts of reasons.

Google trying to improve the world isn’t new. Google Lighthouse has existed for a number of years now, taking over from the likes of YSlow as the defacto performance benchmark, it has caused numerous discussions from the CEO to the Developer in whose ‘version’ is right.

Numerous factors are at play when you load up a website and run a Google Lighthouse which are the main drivers behind Core Web Vitals. Network, Device & Browser Speed & Add-Ons all play a part on the individual machine, but also, website structure, 3rd party response times, CDNs and many other factors all play a part in your performance score. Hundreds if not thousands of variables. Core Web Vitals might be just 3 easy to understand numbers, but the factors that impact them are far-reaching.

RapidSpike is designed with every factor in mind, Google gives you a half-decent picture using all their tools, but it is only half the picture. What good is knowing you have problems, without all the tools to debug it quickly, testing it continuously and consistently, and seeing immediately if the changes you have made are making a difference or your main 3rd party has let you down which has cost you conversions.

If the CEO turns up on Monday ranting and raving that Core Web Vitals are broken and their Google Lighthouse score isn’t what you told them, this can be very disruptive. You need a single source of truth for this data, but Google is too real world to benchmark, there are too many variables. So we thought, let’s do a test on our assumptions, they can’t be wildly different, can they?

We ran 12 tests, we all have Apple Macs of fairly similar ages, we all did it from home, we all did it on the same website all about the same time of day (within an hour). We also added 4 random tests, one fast Windows machine, one slow (& old) windows machine, a Raspberry Pi and a retest 2 weeks later on Mac 1. So what did we find?

Figure 2: Google Lighthouse Test Results
Figure 2: Test Results

Real World Mac Tests:

  • On Desktop between 35 and 86, a margin of error of 51%
  • On Mobile between 10 and 17, a margin of error of 7%.

It was also noted there were fluctuations in other google lighthouse scores. Margin of error 3%.

We ran the exact same tests on RapidSpike, using 12 tests in the same hour, using one location and our machines are rebuilt from scratch every single time for every run.

RapidSpike Lab Tests:

  • On desktop between 61 and 68, a 7% margin of error
  • On mobile between 10 and 13, a 3% margin of error

There were no fluctuations on the other Google Lighthouse scores.

There is a truer version of Core Web Vitals and Google Lighthouse if you run tests continuously and consistently. There will always be fluctuations, because performance isn’t static, but benchmarking a single point of truth, whereby you can accurately judge your performance is incredibly important for the future of website performance.

Register your interest in a free Core Web Vitals Audit today.