Reviewing reviews: 'Top reviews' can help sway shoppers, but there are limits

Although featured—or top—reviews on e-commerce sites can help cut down on information overload for customers trying to make purchasing decisions, too many such top reviews can pose an overload of their own, according to researchers.

In a study of products and product reviews on online retail giant Amazon, the researchers found that top reviews—which are featured reviews that consumers have endorsed as the most helpful—can lead to higher sales and improved customer satisfaction. However, they added that when there are too many featured reviews, their influence can start to wane.

"We found that there is a situation that when there are too many top reviews, you fall right back into the trap where there is just too much information for the consumer," said Wael Jabr, assistant professor of supply chain and information systems in the Penn State Smeal College of Business. "In this natural experiment we had some products with just three top reviews and others with significantly more. For products with way more top reviews, we saw the value of those top reviews goes away."

The researchers used data from about 2.2 million reviews of 1,000 products on Amazon, including all review-related information, such as the overall number of reviews and featured reviews of those products. Sentiment of reviews was determined by the rating the customers gave the product. The researchers also tracked the Amazon sales ranking of the studied products over a 10-month period.

The effectiveness of the top reviews was based on how the performance of individual products changed over time. Specifically, the researchers looked at how the reviews started to disperse in their ratings and how the product sales rank changed.

The study was published in the September issue of Management Information Systems Quarterly.

According to Jabr, e-commerce sites chose to feature reviews because popular products tended to attract numerous reviews. The number of reviews for some of these products can be staggering, he added.

"For example, when Amazon put out the Echo Dot smart speaker, more than a million customers reviewed that product within the first four years of its release," said Jabr. "So, do we need a million reviews to make a good decision on what to buy? Probably not. At a certain point, then, companies started to realize there is an overload that customers will face when we have to navigate this content. Retailers eventually came up with a variety of ways to kind of help you navigate this content, one of which is featuring reviews."

Sentiment match

The researchers also found that the influence of top reviews is strengthened when their opinions tend to match the overall sentiment of the other reviews.

"When you look at the reviews, Amazon shows you the overall ratings of the reviews—for example, how many people gave it a four-star rating, or, how many people gave it a two or three, etcetera," said Jabr. "We wondered, then, if the top review effect can be amplified. And it can. We found that when the distribution of top reviews and the distribution of overall reviews match, then the power of top reviews to influence gains strength. It is almost like there is a confirmation when the top reviews match what the crowd is saying."

The power of top reviews to lift sales and satisfaction is limited, however, said Jabr, who worked with Mohammad Rahman, associate professor of management at the Krannert School of Management at Purdue University. For example, they found that top reviews lack the power to improve the status of less popular products.

The findings could help companies design better webpages while also helping customers make better decisions, said Jabr.

"Platforms, such as Amazon, are, of course, in the business of selling stuff, but selling stuff alone is not enough," said Jabr. "Platforms want consumers satisfied with their purchases—and not return those purchases. They also want repeat consumers. In fact, Jeff Bezos himself is quoted saying, 'We don't make money when we sell things. We make money when we help consumers make better decisions.'"

The study explores whether there is a certain magic number of reviews as being an optimal amount of top reviews, according to the researchers.

"While the natural experiment does not compare every combination of numbers—for example, two reviews compared to three reviews, or two compared to four—we found that products with three reviews faired better than products with a varying number of reviews ranging from four to 10," said Jabr.

In addition to being thoughtful about selecting and displaying top reviews on a webpage, the researchers also suggest that at a certain point, companies should switch from encouraging customers to review products to asking them to endorse reviews.

"Retailers often default to sending you an email saying, 'Please rate our product,' which we think is great," said Jabr. "But when there are enough reviews, they may want to find a way to nudge the customers to decide on top reviews because that's going to be much more valuable then writing one more review."