Revisiting the Reliability of Wine Ratings

I really don't think that people should over-think wine, but occasionally I temporarily stop enjoying wines to think about it. Lately I've been thinking about wine ratings--their reliability, etc.--in light of some ink that has recently been spilled (esp. see here and here). On the one hand, wine ratings are important—they are clearly a desiderata of producers, once a year the major print publications and even minor businesses herald their Top 10 or Top 100 wines and we readers consume them. On the other hand, almost everyone would acknowledge that our impressions of a wine can vary significantly from time to time because of a cluster of internal (e.g., palate fatigue, mood, etc.) and external (e.g., bottle variation, etc.) conditions. Note that in most domains a difference is statistically significant if there is a ≥ 5% change; but on the 100 point Parker scale, wines are automatically attributed 50 points, so the 100 point scale is really a 50 point scale. This means that statistically significant variation on the 100 point scale is ≥ 2.5. I doubt that there are many—if any—people who would doubt that their ratings of the same wine never vary ≥ 2.5 on different occasions. Once consumers (fully) realize that variation between 0 to 2.5 isn’t significant on the 100 point scale, consumers will place less importance in a rating, especially a rating just below the boundary of a multiple of 10, e.g., 89, and a rating just above the boundary of the next multiple of 10, e.g., 90. A wine that scored 90 on one occasion will almost certainly receive a different score on another occasion for one reason or another. I would argue that many consumers have fully realized this. But many critics have not acknowledge this either because they mistakenly believe that their tasting and evaluation abilities are so precise that they would never attribute a score ≥ 2.5 to the same wine on separate occasions; or they mistakenly believe that their professional stature would suffer if they publicly acknowledge this.

In light of typical variation in a wine rating on the 100 point scale, purchasing wines on the basis of a single rating, even of a professional critic (no one shall be named), is… well, irrational. So what should consumers use to guide our purchases? I have two suggestions—neither of which are infallible, but are more reliable than using a single score from a professional critic. First, you can stick with ratings and use the average from multiple ratings provided by professionals, non-professionals or some combination thereof. How many ratings are sufficient, you ask? It’s unclear, but 6 or 7 is a standard range of series to eliminate bias. One popular source for ratings that automatically calculates an average (as well as the median) is CellarTracker! (If you want to a more reliable average rating, you can drop the high and low score, etc.) Second, you can forego ratings altogether and use good old-fashioned word of mouth. That is, find a trustworthy friend who reliably recommends wines that you enjoy. My friend is a clerk at my local wine shop; and her recommendations are reliable even in the face of contrary evidence, e.g., a $10 price tag, etc. One might argue that the scores and tasting notes of a single professional wine critic can be as reliable as the recommendations of your trustworthy friend. But I would argue that because I have more access to my trustworthy friend at the local wine shop, and therefore access to more information, her recommendations are more reliable than those of the professional critic to whom I have no personal acess.