Reading between the lines of Yelp Reviews - I'm a data geek.

Reviews are not always what they appear, especially on Yelp.  This post won't get into the nitty gritty about filtering, shill reviews, and other shenanigans, but this post will give a different view of how you can read between the reviews on Yelp.

*Note: This data generally applies to Austin-area restaurants (trailer or brick and mortar).  This does not apply to other types of businesses or mail order food products.  I have not looked at the data concerning those types of businesses. I have not looked at data in other cities. If you have a restaurant, don't fret if your review behavior doesn't line up with what generally happens.  Your restaurant might just be an outlier.  This will not apply.  

*In less webbie (or techie) locations or places with a smaller population, I find Yelp to be only marginally useful. 

When I use Yelp for search, the first two questions I have are:

• 1. What is the average star rating?
• 2. How many ratings? 

In Austin, you would expect a successful places to have a relatively large number (more than 75) of reviews after a year.  If the location has very few reviews after being open for a year or so, my next questions are:

• 1. Who wrote the reviews?
• 2. Is there anything that makes this place an outlier (difficult to reach location, odd hours, unavailable to the public, special events only, or otherwise)?
• 3.When I Google this location, is there anything odd on the web?  Is the owner an axe-murderer?

After making sure that there was nothing that made this place a horrendous outlier, I would think read the actual reviews for the qualitative data.  This is typically a laborious process just to figure out where I should eat. 


After going through this routine of trying to interpret the data, I found that the review behavior is much more telling than the reviews themselves.  Any place can have a high average star rating as it is fairly easy manipulate.  A restaurant can have only four ratings of five stars each, geee...... that restaurant looks perfect doesn't it?  I would beg to differ, the low number of ratings could be because people really dislike the restaurant and avoid it, or it could be that some Yelp users just don't write about negative experiences.  In any case, a low number of reviews after being open for business for a significant amount of time is a red warning flag.

Being the big data geek that I am, I've looked at review behavior for many restaurants over a long period of time.  One thing that stuck out was that successful restaurant consistently garnered reviews, positive and negative.  Not so successful restaurants would stagnate after an initial burst of reviews, garnering maybe one review every month or so.  In my interpretation of that behavior, a restaurant with a four-star average with a consistently growing number of reviews is a much better choice than a restaurant with perhaps a 4.5 star average, only a handful of reviews, and the latest review was two months ago.  Personally, I would rather eat at a restaurant with a lower average rating that had consistent growth in their reviews rather than a restaurant with a high average rating but no new reviews.  That's just my interpretation of the data, and my opinion. 

I will also add that public relations effort does have an impact on the review behavior, but only significantly in the early days of a restaurant's opening.  There will inevitably be a spurt of reviews (positive or negative) when a restaurant first opens, but if the restaurant fails to perform, the reviews stop coming in.  Additionally, if the restaurant participates in branding and publicity efforts, that might cause other spurts of reviews.  However, it doesn't change the review behavior significantly in the long term.

The chart above is something I created to model the review behavior, and it is based off of many Austin restaurants.  I will not list the ones that are unsuccessful, but I'll point out one that is successful.  The Noble Pig is a great example of a successful restaurant that did not engage in a large public relations effort.  They have been open for only about six months, and they are located a 40 minute drive away from downtown; yet they already have 69 reviews to-date with a 4.5 star rating.  Even though the location makes them an outlier, they are consistently garnering reviews.  That's a place I would definitely visit again, and again, and again.  *Disclosure: I met the chef at several events, and I really like his sandwiches. 

This post is just a suggestion that reviews and star ratings should just be taken with a grain of salt.  Looking at review behavior can give you more in-depth look at a restaurant's success. I'm using the word "success" loosely.  You can define it however you like.

Yelp's New Transparency Bad New for Businesses who write shill reviews

Yelp is trying to turn things around.  Public distrust.  Angry business owners.  Three class-action law suits.  Yelp has been on many people's naughty list lately, and they have implemented something new to turn that all around.  They have allowed previously "filtered" or "suppressed" reviews to be viewed by anyone and everyone.  Here's several links to the details on that, and I'll avoid rehashing all that information.   Yelp's Official Blog Post about Filtering Mashable's Blog post about Filtering However, I spent a large chunk of time studying Austin business reviews, and here's what it means to users and to businesses (from my limited information that is).  I noticed that some businesses has only 10% of reviews filtered while others had more filtered reviews than reviews visible (i.e. 28 reviews visible, 47 filtered).  The percentiles were highly variable.  I haven't seen any patterns to them yet.  A few other things I gathered included: 1. Yelp has increased it's transparency, and that is bad news for sneaky businesses out there.  With the filtered reviews exposed, I easily found many businesses who wrote themselves shill reviews either using their own names or only writing positive reviews for other businesses they owned.  It appears that even if reviews are flagged by the community as being fake (i.e. the master debater's thread) they are either filtered or deleted due to violation of terms of service (TOS).  While I can't be absolutely certain due to the limits of information I have, I think they are deleted and placed below the filtered reviews.  You can't read the reviews anymore, but you can see who wrote them. 2. One thing I do notice is that the filtering system is fairly ineffective.  Sometimes it filters out people with 20 reviews, yet it leaves people with only one review unfiltered. Sometimes it filters out people who have spent a significant amount of time writing a thoughtful review, yet it leaves people with one-line reviews visible on the business page. That's the problem with having an algorithm do your work.  3. The users who have filtered reviews (if everything is truly computer run numbers) should have all their reviews filtered.  I did find this to be true for the few people I researched on Yelp, but I can't be absolutely sure that this is true for all users who have filtered reviews.  Also, on the user(s)' profile page, it does not show whether or not their other reviews have been filtered.  It is also unlikely that any of these users know that their reviews have been filtered.  Theoretically, there should be a waiting period for all new users who have to work their way out of the filter by crossing whatever threshold the algorithm uses.  Yelp does not inform new users about that they are being filtered, which can be good or bad.  New users probably have a difficult time understanding the concept of filtering, and it might scare off new users from joining if they have to read a bunch of rules and regulations.  However, I think that if new Yelp users were informed of this filtering mechanism, Yelp might see more engaged new users.  That's all speculation though.  I can't think of any psychology studies examining commitment to a social community if there was a heavy time investment upfront without immediate reward.  4. It appears that some administratively removed reviews are never shown in the deleted reviews.  I noticed very recently that Yelp had administratively deleted all reviews for a business if they were written before the business officially opened (i.e. Urban An American Grill and Lick it Bite it or Both).  After the business opened, those reviews did not reappear nor did they appear in the filtered or deleted review section.  It is my understanding that Yelp administrators would send an email to each of the users informing them that the review was removed.  However, there have also been some claims about missing reviews without an email from administration.  It does appear that all other administratively removed reviews that were flagged by the community do appear in the deleted review section.   My advice for businesses is to scroll through your filtered reviews.  If they were written by yourself, your employees, or your friends, delete them as soon as possible.  These reviews might have been previously filtered and not visible to the Yelp community.  Now that they are visible to the Yelp community, users might feel as if you were trying to fool them with fake reviews and retaliate.  You might have escaped the master debater's thread previously, but now you can't hide those filtered reviews.  Even if users flag them now as shill reviews, they might still end up in the deleted section.  Deleting them now from the accounts you've (or employees or friends) created is the only way that they will disappear.  Additionally, reviews that are deleted are not searchable and do not come up in Google searches.  Threads, however, have no delete or edit button.  If you are called out on a thread, it is there to stay forever.  *Google can find quotes from filtered reviews though.  Yelp has taken steps to be more transparent, but that also means that the businesses must also be more transparent.  You can run and hide, but not for long.