Title: Measuring Usability: Preference vs Performance Author: Jakob Nielson & Jonathon Levy Journal: Communications of the ACM Issue: April 1994, Vol 37 (4) Pages: 66-75 Title: Usability Engineering in Dollars and Cents Author: Clare-Marie Kabat Journal: IEEE Software Issue: May 1993 Pages: 88-89 Title: Is Usability Engineering really worth it? Author: Jakob Nielson Journal: IEEE Software Issue: November 1993 Pages: 90-92
I hope that some people find these of use. They have been useful for me in my MSc project, which is evaluating the "usability" of textual interfaces for British Steel.
I don't have at my fingertips published studies of how much usability saves. I can recommend that you check the publications of the IEEE Human Factors society and the ACM Special Interest Group on Computer Human Interaction (SIGCHI). They may have the information you want, and if you call their offices they may give you better pointers.
However, that kind of study has not been of much interest to me. That may surprise you, but the reason is because, as in many things associated with good design, "your mileage may vary". How much usability you get out of a given hour of work depends upon how good a designer you use, and how much $$$ value you get is dependent upon costs of development, costs of sales, costs of support, and market preferences and the premiums they can command, and even the ROI to customers. And all that differs from product to product by huge amounts. I don't really care what an industry average is, or some sample test in some laboratory. I want to know what MY value will be.
So I can't give you a magic number that says that if you hire a usability designer you will save $100,000 and profit $1,000,000 more than if you don't. But I can tell you how to figure it out for any particular case. And that *IS* what interests me. I don't care a hoot what the industry average is, but I really do care what benefit my clients will see. So here is how I look at it for them:
$$$ benefits come in the following flavors:
1) Reduced engineering/development costs and earlier time to market
2) Reduced testing/QA costs
3) Reduced Support/Training costs
4) Reduced sales costs and shorter sales cycles
5) Increased Sales Volume (increased customer preference)
6) Increased Price (premium pricing for preferred products)
7) Customer Return on Investment (ROI) from easier, faster or more
accurate performance of their business tasks.
Design for usability can benefit every one of these areas. How much depends upon the skill of the designer, and the total costs of each area at start. Here's how to evaluate the potential benefits. Remember Sutton's Law: Why rob banks? because that's where the money is. In order to ensure usability, we work BACKWARDs through this list:
Designing for usability is all about identifying what the customer's task is, and finding the easiest, fastest, most accurate way for the customer to do that. This turns out to be remarkably easy to measure in real life. Just set up a head to head test. Give two customer prospects the same set of real life problems. Let one use your tool, the other user the tools they already have and know.
Measure how long it takes to do the real world task in each case. Now you have a concrete number for the time saved (for instance in the last test where I did this, my new product user took 1 hour to do the tasks, the control user took 2 hours). You can convert that to a $$$ amount by multiplying by the cost of that employee (typically salary, benefits, overhead, etc.: the "fully burdened" costs. For a programmer that is about $50/hour). So in my example case we saw that the new product could save about $50/usage. How many times is the product used? In my case we estimated that it would be used about once a week, or 13 times a quarter, for a net savings of $650 per quarter or $2,600 per person year in SPEED improvements alone. (note, it is important to test the time of the TASK, not just how fast the computer works. In my case, the computer time wasn't faster, but the TASK time was dramatically so. That is common in Usability).
Now check both for errors. Does the tool make it less like to make errors? More likely to detect and correct them? In our case, for instance we found 5 undetected errors in our control user's work, none in the new product user's work. To convert to $$$, get a measure for the cost of an undetected defect. For instance in our case, Tom DeMarco estimated that a minimum cost for finding and fixing an undetected defect was $1000/defect. So in our case, we saved $5000 in the one usage, That's $65,000/person-quarter or $360,000/person-year saved in our case using the once per week usage levels given above. Again, adjust based upon your own expected usage patterns.
Next, you can compare you people FEEL about using the tool vs. using the alternatives. These are inherently subjective, but then the purchase decision is totally subjective too! This is still useful information, because it relates to the expected increase in the volume in sales, and in the premium you might expect to command in price. You'll of course use your measured ROIs to help promote the product--and when sales results come in you can see if these benefits accrued or not.
Moving back from the Customer, we focus on our own Sales team. How can we reduce the time and effort it takes to sell the product? Having a DESIRABLE product makes it easier of course, but there is more to designing a more usable product than just customer demand. Because sales is a set of tasks and phases too. There is awareness, acquisition, installation, configuration, pilot test, mass deployment. A usable product is better in each of these activities as well.
First, how are people going to become aware of the product. Will the trades promote it as better than competitors? Will there be word of mouth? Is there a simple compelling story that is easy to understand in a short sales brochure, a trade show demo, press release, etc.? All those things make it easier for the customer to become aware of it.
How can they acquire it? Is it available through the mail? At tradeshows? Can they download it electronically, is it already bundled on their system? or on a CD they get for free? All these things affect the ease of acquisition.
Now, how easy is it to install it? Just press one button? or do you have to answer a lot of questions up front? In fact, do you have to KNOW something about what you want, or how the tool behaves to answer those installation and configuration issues?
How can the user pilot test it? Does it automatically run a demo of how to use it? Does it come packaged with a little test suite that you can use to convince yourself of the proposed savings in performance and accuracy? How long does it take to detect real results?
And once the pilot has results, what is involved in converting a whole organization? Can people learn it piece by piece, some people at a time, or does everyone have to come up to speed at once. Usability design addresses all these questions too.
The net result of this is that you can reduce the costs of sales and the length of the sales cycle. Costs of sales are directly measurable, and you can calculate the benefit of a reduced sales cycle by multiplying the reduction in time times the cost of sales people (again fully burdened).
You can also measure support costs. Easy to install products have less support calls (and expense for support staff) than hard to install ones. Same for configuration. Same for learning/training. Same for on going usage. In fact, users won't pay a premium for little used or hard to use features. But they'll call for support on them. The net result is that these LOW USABILITY features cost you more money, instead of generating money.
Let's keep going, now we leave sales and support and go into the development organization. Those low usability features that don't contribute to a bigger price premium still take time to design, to develop and to test. And they will require ongoing maintenance. All for something that doesn't enhance the product. It would be cheaper to never build those features!! And again you measure it directly, just by the time saved by NOT developing the hard to use features, and the time saved NOT testing them. This results in faster time to market too. For instance our last project went from board approval of a concept prototype to product rollout in 45 days. Most other projects take 9 months to a year! And yet our product got a cover notice in a major trade magazine at introduction.
So that's basically how I measure these sorts of things. My customers do too. I'll include an unsolicited testimonial by one of my customers written as an open letter to the net, commenting on their ROI. You can look at the review for my Merge Ahead product in the May 1992 Sun World as well. You'll note that the reviewer gets quantitative at the end: To paraphrase, "If, like me, you [perform the activities that Merge Ahead addresses]... you can expect to save the purchase price within hours of use." Again, a totally quantitative focus.
Ironically, this is old news in the manufacturing industry. It is only new in our industry. Phil Crosby's book "Quality is Free" is well known in manufacturing circles. He showed how designing for Quality actually made manufacturing cheaper. No end of the line inspection stations, no rework. Less cycle time. Exactly the same kinds of things we are talking about here, but applied to manufacturing. Same kind of reduction in engineering costs. And what do we see in manufacturing as a result, fewer parts, fewer things for people to mis-set, misconfigure, adjust, etc. More automation that just does the right thing. That's analogous to creating products with FEWER features, but just the RIGHT features. QUALITY and USABILITY are the same thing in this respect.
There is also about to be printed or just printed a book that addresses this topic:
Bias, R.G. and Mayhew, D.J. Cost-Justifying Usability. Academic Press.