Many people swear by the folks at the Consumers Union who publish, among other titles, the immensely powerful Consumer Reports magazine. At an intellectual level, I like the concept. That the Consumers Union does not accept advertising and pays full retail prices for the products it reviews gives their opinions authority. But I feel like the reviews in Consumer Reports do a poor job of giving consumers good, actionable advice.

The way they review computers — the cover story in their latest issue — particularly irritates me. They spend an inordinate number of column inches on the nitty-gritty technical specifications of each machine, rather than on more understandable metrics. Telling "average" consumers a particular model comes with an Intel Core i7-720QM serves only to confuse them. It's akin listing the specific nozzle configuration of each dishwasher in their kitchen appliance reviews. As with dishwashers — where all that really matters is how well a particular nozzle configuration actually cleans dishes — all that really matters when it comes to reviewing computers is how well it computes. I feel like it would be much more illuminating for Reports to score each model from "Poor" to "Excellent" on more understandable, if less easily quantified, metrics, such as, "Web Browsing," "Cutting-Edge 3D Games," or, "Video Editing."

Moreover, the folks writing in Reports have a tendency to, perhaps implicitly, "over recommend" when it comes to computers. In their latest ranking, the top-of-the-range Apple iMac with a 27" display, which runs nearly $2000 as tested, earned the top score among all-in-one desktop computers. At some level, the machine probably ought to be the top-rated machine. The 27" iMac is indeed very fast; it has a gorgeous, enormous display; and it comes with a lot of consumer features like a wireless keyboard-mouse pair. But I'm sure many people wind up buying that model rather than a similar PC or a smaller (and cheaper) 21" iMac better suited to their needs. I would wager that most consumers would be much happier with a $1500 21" iMac and $500 to spend on a new kitchen television or leaf blower. (This does a good job of illustrating the power of deciding at the margin.)

Likewise, some of their reviews of "chain services" (e.g., chain restaurants, chain hotels) always strike me as rather off. In this month's review of chain hotels, for example, they listed "typical" nightly rates at the Ritz-Carlton as somewhere between $200 and $400, which struck me as low. A quick availability check on the Ritz-Carlton website reveals that the range is appropriate for a basic room, on average: $150 in Dearborn, Michigan compared to $750 in New York City. With services like hotels, I feel like information that accounts for regional differences in living costs would be useful.

The way the hotel ratings are structured also provides a useful lesson in statistics. The review divides the chain hotels into categories on the basis of "fanciness." The Hilton brand falls in the "Luxury" category, while the Radisson brand falls into the "Upscale" category, for instance. And within each category the hotels differ in overall score by very few points, so few, in fact, the differences must be statistically insignificant. Meaning the magazine should scrap their cute table of bars and circles and just say, "Pick a hotel that fits your budget."

(Aside: I was disappointed the hotel review didn't include the Four Seasons. They have a really bizarre management and ownership structure, and I was hoping Reports would confirm the genius of their unorthodox arrangement.)

That said, I do tend to trust the Consumers Union when it comes to cars. In that product category they blow their competitors out of the water. Note that they don't usually spend a whole lot of time discussing the intricacies of vehicles' breaking or steering systems. Instead, they simply provide a nice concise evaluation, using stopping distances — a metric far more easily understood and meaningful than, say, FLOPS, in measuring computational speed — in the case of a car's breaking system. And it's doubtful people will buy "too much car," as they break their car reviews down not by form factor, but by vehicle classes, which tend to correspond quite closely to price and features.

I'm not entirely sure what the "lesson" is in this case. Perhaps that any organization with the aim of helping people make informed decisions should map domain-specific knowledge to metrics meaningful to readers. Think, "great for editing HD video," instead of, "60 gigaFLOPS of computing power."

(Programming note: For those wondering why I haven't posted anything for quite awhile, I'm currently working to migrate my blog to a bespoke Ruby on Rails application. Needless to say, it's taking longer than I had anticipated. Hopefully that will go online late this month. Until then, I'll probably post somewhat more infrequently than usual.)