Abstract: There is a growing concern that e-commerce platforms are amplifying vaccine-misinformation. To investigate, we conduct two-sets of algorithmic audits for vaccine misinformation on the search and recommendation algorithms of Amazon -- world's leading e-retailer. First, we systematically audit search-results belonging to vaccine-related search-queries without logging into the platform -- unpersonalized audits. We find 10.47% of search-results promote misinformative health products. We also observe ranking-bias, with Amazon ranking misinformative search-results higher than debunking search-results. Next, we analyze the effects of personalization due to account-history, where history is built progressively by performing various real-world user-actions, such as clicking a product. We find evidence of filter-bubble effect in Amazon's recommendations; accounts performing actions on misinformative products are presented with more misinformation compared to accounts performing actions on neutral and debunking products. Interestingly, once user clicks on a misinformative product, homepage recommendations become more contaminated compared to when user shows an intention to buy that product.
Introduction: The recent onset of coronavirus pandemic has unleashed a barrage of online health misinformation [4, 22] and renewed focus on the anti-vaccine movement, with anti-vax social media accounts witnessing a 19% increase in their follower base [49]. As scientists work towards creating a vaccine for the disease, health experts worry that vaccine hesitancy could make it difficult to achieve herd immunity against the new virus [3]. Battling health misinformation, especially anti-vaccine misinformation has never been more important.
Statistics show that people increasingly rely on the internet [53], and specifically online search engines [8], for health information including information about medical treatments, immunizations, vaccinations and vaccine-related side effects [6, 23]. Yet, the algorithms powering search engines are not traditionally designed to take into account the credibility and trustworthiness of such information. Search platforms being the primary gateway and reportedly the most trusted source [19], persistent vaccine misinformation on them, can cause serious health ramifications [38]. Thus, there has been a growing interest in empirically investigating search engine results for health misinformation. While multiple studies have performed audits on commercial search engines to investigate problematic behaviour [35, 36, 56], e-commerce platforms have received little to no attention ([11, 59] are two exceptions), despite critics calling e-commerce platforms, like Amazon, a “dystopian” store for hosting anti-vaccine books [17]. Amazon specifically has faced criticism from several technology critics for not regulating health-related products on its platform [5, 55]. Consider the most recent instance. Several medically unverified products for coronavirus treatment, like prayer healing, herbal treatments and antiviral vitamin supplements proliferated Amazon [18, 28], so much so that the company had to remove 1 million fake products after several instances of such treatments were reported by the media [22]. The scale of the problematic content suggests that Amazon could be a great enabler of misinformation, especially health misinformation. It not only hosts problematic health-related content but its recommendation algorithms drive engagement by pushing potentially dubious health products to users of the system [27, 59]. Thus, in this paper we investigate Amazon—world’s leading e-retailer—for most critical form of health misinformation—vaccine misinformation.
What is the amount of misinformation present in Amazon’s search results and recommendations? How does personalization due to user history built progressively by performing real-world user actions, such as clicking or browsing certain products, impact the amount of misinformation returned in subsequent search results and recommendations? In this paper, we dabble into these questions. We conduct 2 sets of systematic audit experiments: Unpersonalized audit and Personalized audit. In the Unpersonalized audit, we adopt Information Retrieval metrics from prior work [42] to determine the amount of health misinformation users are exposed to when searching for vaccine-related queries. In particular, we examine search-results of 48 search queries belonging to 10 popular vaccinerelated topics like ‘hpv vaccine’, ‘immunization’, ‘MMR vaccine and autism’, etc. We collect search results without logging in to Amazon to eliminate the influence of personalization. To gain indepth insights about the platform’s searching and sorting algorithm, our Unpersonalized audits ran for 15 consecutive days, sorting the search results across 5 different Amazon filters each day: “featured”, “price low to high”, “price high to low”, “average customer review” and “newest arrivals”. The first audit resulted in 36,000 search results and 16,815 product page recommendations which we later annotated for their stance on health misinformation—promoting, neutral or debunking.
In our second set of audit—Personalized audit, we determine the impact of personalization due to user history on the amount of health misinformation returned in search results, recommendations and auto-complete suggestions. User history is built progressively over 7 days by performing several real-world actions, such as “search” , “search + click”, “search + click + add to cart”, “search + click + mark top-rated all positive review as helpful”, “follow contributor” and “search on third party website” ( Google.com in our case) . We collect several Amazon components in our Personalized audit, like homepages, product pages, pre-purchase pages, search results, etc. Our audits reveal that Amazon hosts a plethora of health misinformative products belonging to several categories, including Books, Kindle eBooks, Amazon Fashion (e.g. apparel, t-shirt, etc.) and Health & Personal care items (e.g. dietary supplements). We also establish the presence of a filter-bubble effect in Amazon’s recommendations, where recommendations of misinformative health products contain more health misinformation.
Below we present our formal research questions, key findings, contributions and implication of this study along with ethical considerations taken for conducting platform audits.