Automa'c Iden'fica'on of Fake On-‐line Reviews Claire Cardie Department of Computer Science Department of Informa'on Science Cornell University, Ithaca, NY OE, Cardie & Hancock [ACL 2011, WWW 2012, NAACL 2013]
Online Reviews • Consumers increasingly rate, review and research products online • Poten'al for opinion spam – Disrup've opinion spam – Decep've opinion spam
Online Reviews • Consumers increasingly rate, review and research products online • Poten'al for opinion spam – Disrup've opinion spam – Decep've opinion spam
Online Reviews • Consumers increasingly rate, review and research products online • Poten'al for opinion spam – Disrup've opinion spam – Decep've opinion spam
Which of these two hotel reviews is decep%ve opinion spam?
Which of these two hotel reviews is decep%ve opinion spam?
Which of these two hotel reviews is decep%ve opinion spam?
Answer:
Previous Work • Jindal & Liu (2008) – Opinion spam is different from e-‐mail or Web spam – No gold standard decep%ve reviews – Iden'fy duplicate vs. non-‐duplicate reviews
• Mihalcea & Strapparava (2009), Zhou et al. (2004, 2008) – N-‐gram-‐based features, small corpora
• Different decep'on tasks ACL (2011), WWW (2012)
Our Approach • Automa'c methods – Text classifica'on FAKE/TRUTHFUL reviews [features + label] ML Algorithm (novel) reviews [features]
Classifier
label
Overview • • • •
Mo'va'on and Background Gathering Data Human Performance Classifier Construc'on and Performance
Data: Decep've Reviews • Label exis'ng reviews – Can’t manually do this – Duplicate detec'on (Jindal and Liu, 2008)
• Create new reviews – Mechanical Turk
Data • Mechanical Turk – Have: 20 chosen hotels – Want: 20 deceptive positive reviews / hotel – Offer: $1 / review – Get: 400 reviews
Data • Mechanical Turk – Have: 20 chosen hotels – Want: 20 deceptive positive reviews / hotel – Offer: $1 / review – Get: 400 reviews
Data • Mechanical Turk – Have: 20 chosen hotels – Want: 20 deceptive positive reviews / hotel – Offer: $1 / review – Get: 400 reviews
Instruc'ons  Assume that you work for the hotel’s marketing department, and pretend that your boss wants you to write a fake review (as if you were a customer) to be posted on a travel review website; additionally, the review needs to sound realistic and portray the hotel in a positive light.!
Data • Allow only a single submission per Turker • Restrict our task to Turkers – Who are located in the United States – Who maintain an approval ra'ng of at least 90%
• Check for plagiarism
Data: Truthful Reviews • Mine all TripAdvisor.com reviews – From the 20 most-‐reviewed Chicago hotels (6,977) – Discard non-‐5-‐star reviews (3,130) – Exclude reviews wriEen by first-‐'me reviewers (1,607), under 150 characters, non-‐English. 2124 reviews lel.
• Select 400 reviews such that the lengths are distributed similarly to the decep've reviews
Valida'ng the Decep've Reviews • Measure human performance • Can also serve as a baseline
Human Performance
• 80 truthful and 80 decep've reviews • 3 undergraduate judges – Truth bias
Human Performance Performed at chance (p-value = 0.1)
Performed at chance (p-value = 0.5)
• 80 truthful and 80 decep've reviews • 3 undergraduate judges – Truth bias
Human Performance
• 80 truthful and 80 decep've reviews • 3 undergraduate judges – Truth bias
Classified fewer than 12% of opinions as deceptive!
Overview • • • •
Mo'va'on and Background Gathering Data Human Performance Classifier Construc'on and Performance
Classifier • Linear SVM (Support Vector Machine)
Feature representa'on • Three feature sets encode poten'ally complementary framings – Problem in genre iden'fica'on – Instance of psycholinguis'c decep'on detec'on – Standard text categoriza'on
Features: text categoriza'on • Features – n-‐grams
TRIGRAMS+ e.g., the_room_was BIGRAMS+ e.g., the_room
UNIGRAMS e.g., the
Performance
89.6 89.8
Outperform all other methods
Analysis • LIWC+BIGRAMS feature analysis – Spa'al difficul'es (Vrij et al., 2009) – Psychological distancing (Newman et al., 2003)
Analysis • LIWC+BIGRAMS feature analysis – Spa'al difficul'es (Vrij et al., 2009) – Psychological distancing (Newman et al., 2003)
Analysis • LIWC+BIGRAMS feature analysis – Spa'al difficul'es (Vrij et al., 2009) – Psychological distancing (Newman et al., 2003)
Conclusions • People are not good at detec'ng fake on-‐line reviews • Developed automated classifier capable of nearly 90% accuracy when detec'ng (this one type of!) decep've opinion spam
Follow-‐up and Ongoing Work Prevalence of opinions [WWW 2012] Nega've opinions [NAACL 2013] Other domains (e.g., restaurants, doctors) Vary context of decep'on (e.g., domain experts vs. turkers) • Countermeasures • • • •
Acknowledgments Joint work with Jeffrey Hancock (IS, Comm) Myle Ott (CS, PhD student) Yejin Choi (CS PhD student à Asst. Prof., SUNY Stonybrook)
hEp://reviewskep'c.com/