Claire Cardie (Cornell Univ.): Automatic Identification of Fake Online Reviews

Automa'c Iden'fica'on of Fake On-­‐line Reviews Claire Cardie Department of Computer Science Department of Informa'on Science Cornell University, Ithaca, NY

OE, Cardie & Hancock [ACL 2011, WWW 2012, NAACL 2013]

Online Reviews • Consumers increasingly rate, review and research products online • Poten'al for opinion spam – Disrup've opinion spam – Decep've opinion spam

Online Reviews • Consumers increasingly rate, review and research products online • Poten'al for opinion spam – Disrup've opinion spam – Decep've opinion spam

Online Reviews • Consumers increasingly rate, review and research products online • Poten'al for opinion spam – Disrup've opinion spam – Decep've opinion spam

Which of these two hotel reviews is decep%ve opinion spam?

Which of these two hotel reviews is decep%ve opinion spam?

Which of these two hotel reviews is decep%ve opinion spam?

Answer:

Previous Work • Jindal & Liu (2008) – Opinion spam is different from e-­‐mail or Web spam – No gold standard decep%ve reviews – Iden'fy duplicate vs. non-­‐duplicate reviews

• Mihalcea & Strapparava (2009), Zhou et al. (2004, 2008) – N-­‐gram-­‐based features, small corpora

• Different decep'on tasks ACL (2011), WWW (2012)

Our Approach • Automa'c methods – Text classifica'on FAKE/TRUTHFUL reviews [features + label] ML Algorithm (novel) reviews [features]

Classifier

label

Overview • • • •

Mo'va'on and Background Gathering Data Human Performance Classifier Construc'on and Performance

Data: Decep've Reviews • Label exis'ng reviews – Can’t manually do this – Duplicate detec'on (Jindal and Liu, 2008)

• Create new reviews – Mechanical Turk

Data • Mechanical Turk – Have: 20 chosen hotels – Want: 20 deceptive positive reviews / hotel – Offer: $1 / review – Get: 400 reviews

Data • Mechanical Turk – Have: 20 chosen hotels – Want: 20 deceptive positive reviews / hotel – Offer: $1 / review – Get: 400 reviews

Data • Mechanical Turk – Have: 20 chosen hotels – Want: 20 deceptive positive reviews / hotel – Offer: $1 / review – Get: 400 reviews

Instruc'ons  Assume that you work for the hotel’s marketing department, and pretend that your boss wants you to write a fake review (as if you were a customer) to be posted on a travel review website; additionally, the review needs to sound realistic and portray the hotel in a positive light.!

Data • Allow only a single submission per Turker • Restrict our task to Turkers – Who are located in the United States – Who maintain an approval ra'ng of at least 90%

• Check for plagiarism

Data: Truthful Reviews • Mine all TripAdvisor.com reviews – From the 20 most-­‐reviewed Chicago hotels (6,977) – Discard non-­‐5-­‐star reviews (3,130) – Exclude reviews wriEen by first-­‐'me reviewers (1,607), under 150 characters, non-­‐English. 2124 reviews lel.

• Select 400 reviews such that the lengths are distributed similarly to the decep've reviews

Valida'ng the Decep've Reviews • Measure human performance • Can also serve as a baseline

Human Performance

• 80 truthful and 80 decep've reviews • 3 undergraduate judges – Truth bias

Human Performance Performed at chance

(p-value = 0.1)

Performed at chance

(p-value = 0.5)

• 80 truthful and 80 decep've reviews • 3 undergraduate judges – Truth bias

Human Performance

• 80 truthful and 80 decep've reviews • 3 undergraduate judges – Truth bias

Classified fewer than 12% of opinions as deceptive!

Overview • • • •

Mo'va'on and Background Gathering Data Human Performance Classifier Construc'on and Performance

Classifier • Linear SVM (Support Vector Machine)

Feature representa'on • Three feature sets encode poten'ally complementary framings – Problem in genre iden'fica'on – Instance of psycholinguis'c decep'on detec'on – Standard text categoriza'on

Features: text categoriza'on • Features – n-­‐grams

TRIGRAMS+ e.g., the_room_was BIGRAMS+ e.g., the_room

UNIGRAMS e.g., the

Performance

89.6

89.8

Outperform all other methods

Analysis • LIWC+BIGRAMS feature analysis – Spa'al difficul'es (Vrij et al., 2009) – Psychological distancing (Newman et al., 2003)

Analysis • LIWC+BIGRAMS feature analysis – Spa'al difficul'es (Vrij et al., 2009) – Psychological distancing (Newman et al., 2003)

Analysis • LIWC+BIGRAMS feature analysis – Spa'al difficul'es (Vrij et al., 2009) – Psychological distancing (Newman et al., 2003)

Conclusions • People are not good at detec'ng fake on-­‐line reviews • Developed automated classifier capable of nearly 90% accuracy when detec'ng (this one type of!) decep've opinion spam

Follow-­‐up and Ongoing Work Prevalence of opinions [WWW 2012] Nega've opinions [NAACL 2013] Other domains (e.g., restaurants, doctors) Vary context of decep'on (e.g., domain experts vs. turkers) • Countermeasures • • • •

Acknowledgments Joint work with Jeffrey Hancock (IS, Comm) Myle Ott (CS, PhD student) Yejin Choi (CS PhD student à Asst. Prof., SUNY Stonybrook)

hEp://reviewskep'c.com/