Not all ‘review bombing’ is bad for business

For a business on the receiving end of “review bombs”—the sudden influx of online customer reviews following a political or cultural controversy—an interventionist approach to content moderation might seem like a prudent strategy.

But a new open-access study by a Rutgers researcher finds that when review platforms such as Yelp enact tough moderation policies in a bid to sanitize political speech, it can unnecessarily constrain reasonable opinions and cultural context that consumers depend on to decide where to spend their money.

The paper is published in the journal Big Data & Society.

“Simply put, everything you think you know about review bombing is wrong,” said Will B. Payne, assistant professor of geographic information science at Rutgers’ Edward J. Bloustein School of Planning and Public Policy and author of the study.

Online reviews can have a significant impact on an independent business‘s revenue, particularly those on Yelp, the leading local review platform in the United States. One study found that a one-star increase in the average Yelp rating causes a 5% to 9% increase in revenue for nonchain restaurants.

To understand the geographic reach of review bombing incidents and how platforms define acceptable speech, Payne assessed Yelp’s moderation of comments on U.S. businesses embroiled in political controversies between 2004 and 2021.

First, he created a database of businesses affected by national and local politics. Using news sources to identify specific cases and date ranges, he built a dataset of tens of thousands of political-themed reviews. Topics included the 2016 and 2020 U.S. elections, the Black Lives Matter and #MeToo movements and the COVID-19 pandemic.

Next, he analyzed Yelp’s publicly available metadata for reviews of affected businesses, including review date, username, star rating and user location.

Payne then selected two businesses with large numbers of Yelp reviews for in-depth analysis: Washington, D.C.-based pizzeria Comet Ping Pong (subject of the Pizzagate conspiracy theory in 2016) and St. Louis-based Pi Pizzeria, whose owner, Chris Sommers, became the target of online and offline harassment by pro-police supporters after he publicly backed the Black Lives Matter movement in 2017.

In Comet Ping Pong’s case, Payne found that review bombing resulted in primarily negative comments by reviewers mostly on the West Coast—thousands of miles away from the restaurant—while Pi Pizzeria experienced a much more local pattern (largely from the St. Louis area), with an even split of supporters and detractors.

Payne found that Yelp’s automated and human review filtering systems largely responded the same way to each incident, but with considerably different effects. For Comet Ping Pong, of the 283 reviews flagged and removed by Yelp, 229 were negative one-star reviews. By contrast, of the 588 Pi Pizzeria reviews that Yelp removed, most were in support of Sommers’s actions, positive reviews that averaged close to the restaurant’s four-star rating of Yelp-approved reviews.

“Local customers were censored for simply thanking Chris Sommers for standing with them as they marched against police violence,” Payne said. “They weren’t fake reviews about a conspiracy theory; they were legitimate statements by people supporting a business, in this case for the support its owner gave to the neighborhood.”

Payne also looked at Google’s approach to content moderation and found that unlike Yelp, Google rarely removes politically themed reviews. This, too, can be a double-edged sword; Comet Ping Pong still has dozens of public Google reviews referencing the false Pizzagate conspiracy.

The data does have several limitations, Payne said. First is the possibility that the self-reported location of Yelp users was inaccurate, or that some users could have moved between the time they set up their Yelp profile and when they wrote a review.

Additionally, reviews on Google Maps—a popular Yelp competitor—don’t contain user location information and can be removed by Google without leaving the public metadata traces that Yelp provides for transparency.

As review bombing continues to test review platforms’ approaches to political discourse—the most recent example surfaced this month, when Yelp halted reviews of a McDonald’s franchise in Feasterville, Penn., where former President Donald J. Trump had held a campaign event—Payne said it’s worth considering whether content moderation has gone too far.

The question is particularly relevant for Yelp, which has used corporate communications and review search filters to support Black-owned, women-owned, and LGBTQ-inclusive businesses—speech that isn’t permitted by reviewers themselves unless accompanying a customer experience review.

“Having a one-size-fits-all, review bombing or political speech policy can lead to the suppression of legitimate expressions of support for the role a small business plays in the community, as in the case of Pi Pizzeria,” Payne said. “Some might disagree that the political positions of a business owner should guide consumer behavior, but on Yelp, it’s a choice that users can’t even make for themselves.”