The Perception of Performance

Perception is a mix of a user’s expectations, usability, and performance. A well designed solution can get high user satisfaction despite some annoying delays and a poorly designed solution can be perceived as slow despite its fast technical speed.

When performance is perceived to be better than expectation, satisfaction is high. Conversely, when performance is perceived to be below expectation, satisfaction is low.

This is known as Maister’s First Law of Service:

“Satisfaction is the difference between what was perceived and what was expected.”

What sets users’ expectations?

All durations are meaningless without a reference for comparison. Without some reference, such as prior or similar experiences, perception will be meaningless. What is established in memory as an expectation will set the tolerance threshold for an interaction, beyond this threshold perceived durations will be judged as slow.

An important point to consider is that user expectations can be established by any source, with more consideration for general popularity than science or engineering.

So when Google says that all sites should load under one second, this actually influences users’ expectations, and as these statements spread, these numbers have a high probability of becoming the expected standard. As such, it becomes highly important to identify what the current standards are, how they are determined, how they are achieved, and, if necessary, what corrective means you need to take to get there.

Because perception is highly susceptible to distortion, what users perceive may be quite different from what is your real performance.

Does UX influence performance?

In 2001, Christine Perfetti and Lori Landesman published a study that aimed to check if there was a strong relationship between page download time and usability. Their assumptions were that websites with faster download times would be considered more usable and rate higher than slower websites, and that the ratings would correlate strongly with the actual speed of the websites.

When they analyzed the results, the researchers found that there was no correlation between the download speed and the website that users’ perceived to be faster. about.com was rated the slowest website of the study, but in fact it was actually the fastest, having an average load time of 8s (remember this was 2001). Amazon.com was rated the fastest by users, but had the slowest download time with 36s (yes this really was 2001!).

What the study found, however, was a strong correlation between perceived download time and task completion. When users are able to complete tasks, they perceive download time as fast. Conversely, if users can’t find what they’re looking for on a website they’ll regard it as slow and a waste of time.

Is faster always faster?

So let’s say that a talented developer on your team found a way to bring your website load time down from 8s to 6s, but it will take two weeks of code optimization. As 2s is a substantial improvement, you consider that the effort is worth the time and effort, but is it really?

How users perceive time

Many of the measurements we do in life are done not with objective physical instruments but with subjective mental approximation.

Understanding the users’ ability to detect timing differences is of major importance when defining performance objectives. You need to be able to know if the objective you’re aiming for is even perceivable, otherwise the ROI (return on investment) of your efforts will be highly questionable.

On the other hand, it’s also important to know how much performance degradation is allowed until users start to perceive it. This is referred to as regression allowance.

Weber-Fechner Law

Weber’s Law, which later evolved into the Weber-Fechner Law involves a key concept called just-noticeable difference, typically referred to as ‘jnd’.

jnd is the minimum increase or decrease in magnitude of a property of a stimulus (the brightness of a light bulb, the volume of a static buzz, etc.) that is detectable or, as the name implies, noticeable.

The 20% rule

The main concepts of Weber’s Law and jnd have a direct application in human-computer interaction and performance. Based on data from human timing research, a good rule of thumb is to use a ratio of 20% of the duration in question.

To put things in simple terms, to create a noticeable performance improvement, that is perceived by your users as such, that improvement has to be of at least 20%. So if your page loads in 10 seconds, making it load in 8 seconds or less would be a noticeable improvement, whereas 9 seconds wouldn’t make much difference. Conversely, if your page loads in 4s the performance degradation allowed is 0.8s.

Wanna go faster than the competition?

The 20% rule can also be applied to the metrics of your competitors, to determine the point at which users will perceive your site as faster or the point at which users don’t perceive that your website is actually a little bit slower.

Sometimes you just can’t go faster

Sometimes you just can’t go faster than the competition because of an insurmountable list of constraints you can’t get over, but you need to eliminate the differentiation, whether by catching up to your competitors’ performance or to a market standard you have fallen short of. In this cases your desired outcome is to neutralize the differentiation or to achieve the ‘not by much standard’.

To stay in the game you need to deliver performance that is on par with, or at least close to, your competitors so that users perceive that neither product has substantially more value than the other.

So if your current website loads in 10s and your competitor’s website is at 4s, how do you set your new goal? More specifically, what new objective should you set to neutralize the differentiation?

According to research in human timing, your objective is at about 6 seconds. How do you get to this value? By calculating the geometric mean of the two values, 4 and 10, which is just the square root of their product, that is √(4*10).

Research says that beyond the geometric mean, the probability of associating a value with the higher value of the two increases. That is, values above 6 will be perceived more like 10 and values below 6 will be judged to be closer to 4.

Therefore the geometric mean would allow your product to stay in the game, the lower you can go from there the better.

Conclusion

In closing, I think this formula used in a talk by Ilya Grigorik sums this up nicely:

Perceived performance = f(Expected Performance, UX, Actual Performance)

Frequently Asked Questions on Perception of Performance