6 Comments“The Sky Is falling!” shouted Chicken Little. Perhaps you’ve heard the line before, but if not, a short recap of the children’s story: Chicken Little gets the whole forest into a tizzy, thinking that the sky is falling, though unbeknownst to Chicken Little, it was only an acorn that hit him. Everyone follows him, until they end up in the fox’s hole… only to never make it out again.
The moral of the story here is: don’t jump to immediate conclusions and assume calamity has struck.
The True Effect of Amazon’s TOS Changes
On October 3rd of this year, the community of Amazon sellers were thrown for a loop with a change in the Terms of Service that banned incentivized reviews.
Was this change just an inconsequential acorn that fell, or was the sky actually falling for Amazon sellers? Anecdotal evidence zoomed through the seller community with varying degrees of reliability.
At Review Kick, thousands of sellers who were presumably effected by this change in Amazon’s Terms. We recently gathered the data and ran rigorous analysis, to determine exactly how the change in Amazon policies effected sellers.
TL; DR: The change in Terms of Service ultimately leveled the playing field, to remove sellers who were “gaming the system”, by quickly generating lots of reviews. Sellers who had a sudden surge in reviews lost the most reviews.
The data set we worked with
Of almost 6k products….
The foundation of our study is based on a subset of the products that were promoted on Review Kick. In total, there were 7967 that we tracked.
We had data for 5995 of the products. This means that we had data for these 5995 products before and after the October 3rd TOS change, including the Category, Number of Reviews, and Rating.
…that were in these categories…
The category that was most represented in our data set was Home & Kitchen.
The breakdown of the products in our dataset, by category, can be seen here:
Note that this graph only includes categories that had more than 10 products, so you won’t see some categories like Video Games. Furthermore, note that this is on a logarithmic scale, so the bars looked more compressed as they move to the right.
…roughly 25% of products that we were tracking lost reviews…
Of these 5995 products, 1548 lost reviews. Thus, 25.8% of our random sample lost reviews. This is a statistically significant result (ie not a random occurrence of luck).
The categories that were impacted by the TOS
We were curious to see which categories, if any, were most impacted by the Terms of Service.
Looking to see how many products lost reviews, by category, we get the following graph:
This graph shows that Kitchen & Dining and Electronics were outliers in how many products lost reviews.
This is very interesting data to see!
But I was curious, if they lost reviews, how many reviews did they actually lose, as a percentage and as a number?
For example, even if almost 65% of products in Kitchen & Dining lost reviews, it wouldn’t be so bad for sellers if it were only 1 or 2 reviews.
Loss of reviews in % and #
We plotted out this data, to see how the number of reviews correlated to the percentage of reviews. Did sellers that lost a bunch of reviews after the TOS lose a large percentage of their reviews? Or were they just well-established products with a ton of reviews, and losing 100 reviews was just a drop in the bucket for them?
Here is a scatter plot that shows the reviews lost, in percentage and number:
Let me explain what we’re looking at here.
In the graph above, each product that lost reviews is shown as a dot. On the x-axis are the number of reviews the product lost, and on the y-axis is the percentage of reviews that were lost.
We can see along the top of the graph that 9 products lost 100% of their reviews and on the right side of the graph we see that 5 products lost more than 2000 reviews.
The blue trend line highlights that products which lost more reviews tended to lose a higher percentage or their reviews. Additionally, if a product lost more than 200 reviews, that tended to represent about 2/3rds of their reviews on average.
Percentage of reviews lost by product
Looking at this in another way, let’s look at the percentage of reviews that products lost overall. Here’s the results in this chart:
In the graph above, we see the percentage of reviews that products lost. For example, 210 products lost a maximum of 10% or their reviews, and 25 products lost between 90% and 100%.
What this chart doesn’t show is that among products that lost reviews, the median was 28 reviews or 41% of their reviews.
Note that we used the median because some lost 2000, which is an extreme amount that would skew the data if we used an average. In many ways, the median (as described by the middle figure) negates any outliers, so is most representative.
“Among products that lost reviews, the median was 28 reviews or 41% of their reviews.”
Were any particular sellers/categories targeted?
I also wanted to see if we could identify any pattern in how Amazon enforced its Terms of Service change. For example, were there any types of products, categories, larger/smaller sellers or price points that lost an outsize number of reviews?
To answer this question, we compared the number of reviews before the TOS change, against percentage of reviews lost. The scatter plot looks something like this:
As the blue trend line shows in graph above, products that had a lot of reviews before the TOS change tended to lose a slightly higher percentage their reviews. However, the difference was not very large. No particular categories have been targeted for removal.
“We saw no categories targeted for removal of reviews after the TOS change.”
What was the overall change in rating and reviews for those products that lost reviews?
Among products that lost reviews and saw a reduced rating, their median loss in star rating was just 0.13, or about 3%.
What does this all mean?
This study confirmed many of the suspicions and hypotheses that I had going into this. As it turns out, a majority of sellers were not impacted—our data shows 26% of products lost reviews.
Moreover, it seems that there was a pattern that the products with more reviews had a greater likelihood of having reviews removed. Additionally, these products had a higher percentage of these reviews removed.
I interpret this to mean that Amazon targeted products had aggressive spikes in product reviews. This was fully legal and within Amazon’s Terms of Service, but is no longer a viable strategy after the TOS.
Ultimately, Amazon set up a seller environment that removes any unfair advantage of using reviews services. This will prevent stop any sellers that were “gaming the system” to boost their reviews, organic ranking, and of course, their sales.
The benefits of Amazon prohibiting incentivized reviews
It’s not all doom and gloom though. In fact, prohibiting incentivized reviews is a great move by Amazon. Here’s why:
#1 Amazon success relies on fundamentals once again:
It became clear that some sellers were abusing the reviews system, and getting lots of reviews in a very short time. Moreover, the products that had many reviews naturally rose to the top of the organic rankings for main keywords, and made it intimidating and difficult for new products to enter the niche. Ultimately, this did a disservice to Amazon. New sellers did not enter the market, and Amazon customers were denied the opportunity of choice.
#2 Amazon is no longer a first-come, first-serve game:
Related to the point above, there is opportunity again for new sellers to enter product niches where otherwise it would not have been a prudent decision before the TOS change. I think this will benefit Amazon customers, by having access to a greater variety (and presumably better quality products).
Finding success and making money on Amazon has shifted towards the long-game. This means sellers can no longer launch a product, collect a bunch of reviews, and shoot to the top of the organic rankings. I was certainly a beneficiary of this with Jungle Stix. However, times have changed and success will now rely on the following digital marketing fundamentals:
- Product research
- Search engine optimization
- Pay per click expertise
- Conversion rate optimization
- Email marketing
#3 Customer Trust Is Restored:
Despite being legitimate, honest reviews that were generated in exchange for discounts skewed the overall rating of the product to be higher than it would without incentivized reviews. We see this with the drop in rating of 0.13 stars for products that lost reviews after the TOS change.
I believe Amazon’s main priority was to restore customer trust in its product reviews system. As we know, social proof and reviews are critical in creating a marketplace of trust. Therefore, the change will restore customer trust and increase the likelihood of people purchasing products on Amazon vs. in brick and mortar stores. This is especially important for private label sellers, who do not have a brand reputation to help convince a customer to purchase their product.
Customer trust and social proof is vital for private label sellers, who do not have the brand reputation to help convince a customer to purchase, compared to brick and mortar stores.
What now?
Here is my conclusion: it’s time to get back to work!
Seriously though, I think Amazon just opened up the opportunity for new sellers to enter the market and differentiate their offering, whether it is by selling a different set of features, a higher quality product, better listing, etc.
Notes About This Study
If you are a real data geek, we have a summary of our data queries for you to DOWNLOAD HERE. I worked with our Data Scientist, Ellis, to pull these insights. By the way, we carried out this work in RStudio, a powerful open source and free program to do deep statistical and graphical analysis.
_______________
As with any data study, there are some limitations that are important to note. A few of which are:
- This study was comprised of only products that were listed on Review Kick. This includes private label sellers, wholesalers, retailers, and retail arbitrage sellers.
- We did not track and include the content of the reviews. Therefore, we can not correlate the reviews that were removed by Amazon and whether they included a disclaimer (noting that the product was received at a discount in exchange for a review).
- We only took snapshots at two points in time, on October 2nd (before the Amazon TOS change) and on December 16th (two and a half months after the TOS change). It would have been interesting to track how fast the reviews were accumulated before the TOS change, to determine if there was a correlation between a spike in reviews and a subsequent loss in reviews afterwards. Alas, that will have to come with the next study.
6 comments on “The Surprising Results of A Data-Driven Study on Amazon’s Terms of Service Change”