An Analysis of Credibility of Content on Twitter

GRD Journals- Global Research and Development Journal for Engineering | Volume 2 | Issue 5 | April 2017 ISSN: 2455-5703

An Analysis of Credibility of Content on Twitter Dr. Bhoomi Gupta Department of Computer Science and Engineering Maharaja Agrasen Institute of Technology

Shivani Shukla Department of Computer Science and Engineering Maharaja Agrasen Institute of Technology

Abstract We analyze the information credibility of news propagated through Twitter, a popular microblogging service. Previous research has shown that most of the messages posted on Twitter are truthful, but the service is also used to spread misinformation and false rumors, often unintentionally. Specifically, we analyze microblog postings related to “trending” topics, and classify them as credible or not credible, based on features extracted from them. We use features from users’ posting and re-posting (“re-tweeting”) behavior, from the text of the posts, and from citations to external sources. Our results show that there are measurable differences in the way messages propagate, that can be used to classify them automatically as credible or not credible.

Keywords- Twitter, Credibility, Online Social Media

I. INTRODUCTION With the evolution of online social networking and micro-blogging mediums, two major changes have occurred in the landscape of the Internet usage firstly, the Internet is replacing traditional media like television and print media as a source for obtaining news and information about current events; secondly, the Internet has provided a platform for common people to share information and express their opinions. Quick response time and high connectivity speed have fueled the propagation and dissemination of information, by users on online social media services like Facebook, Twitter, and YouTube. Work presented in this paper primarily focuses on Twitter; Twitter is a micro-blogging service, which has gained popularity as a major news source and information dissemination agent over last few years.

Users on Twitter, create their public / private profile and post messages (also referred as tweets or statuses) via the profile.

The maximum length of the tweet can be 140 characters. Each post on Twitter is characterized by two main components: the tweet (content and associated metadata) and the user (source) who posted the tweet.

Though a large volume of content is posted on Twitter, not all information is trustworthy or useful in providing information about the event. The credibility and quality of information often plays a critical role during high impact events. Fake news and rumors also propagate along with genuine news Researchers have shown that role of Twitter during mass convergence and emergency events increases considerably than more general Twitter activity. They showed that tweets during such events led to more information broadcasting and brokerage. This was the motivation for us to specifically consider some major events during 2016-17 for our analysis.

The organization of the paper is as follows: Introduction is provided in Section I, Literature Survey is mentioned in Section II, while Section III gives the Research Methodology. Section IV discusses the analysis. The last section i.e. Section V presents the conclusions and future work.

II. LITERATURE SURVEY The literature on information credibility is extensive, so in this section our coverage of it is by no means complete. We just provide an outline of the research that is most closely related to ours.

Among online news sites, blogs are considered less trustworthy than traditional news sites [4]. A survey in 2005 showed that, even among young people, blogs are seen as significantly less trustworthy than traditional news sites. An exception seem to be users with political interests, which rate the credibility of blogs sites high, particularly when they are themselves heavy blog users[6].

While most messages on Twitter are conversation and chatter, people also use it to share relevant information and to report news [7, 8, 9]. Indeed, the majority of “trending topics” –keywords that experiment a sharp increase in frequency– can be considered “headline news or persistent news” [10]. The fact that Twitter echoes news stories from traditional media can be exploited to use Twitter, e.g. to track epidemics [11], detect news events [12], geolocate such events [13] ,and find controversial emerging controversial topics. Recently Mathioudakis and Koudas [15] described an on-line monitoring system to perform trend detection over the Twitter stream. In this paper we assume that a system for trend detection exists [15] and focus on the issues related to labeling those trends or events. Twitter has been used widely during emergency situations. Journalists have hailed the immediacy of the service which allowed “to report breaking news quickly – in many cases, more rapidly than most mainstream media outlets”[17].

All rights reserved by www.grdjournals.com

98

An Analysis of Credibility of Content on Twitter (GRDJE/ Volume 2 / Issue 5 / 016)

Presence of spam, compromised accounts, malware, and phishing attacks are major concerns with respect to the quality of information on Twitter. Techniques to filter out spam/ phishing on Twitter have been studied and various effective solutions have been proposed [1].

Truthy , was developed by Ratkiewicz et al. to study information distribution on Twitter and compute a trustworthiness score for a public stream of micro-blogging updates related to an event to detect political smear, misinformation, and other forms of social pollution[20]. In their work, they presented certain cases of abusive behavior by Twitter users. Castillo et al. showed that automated classification techniques can be used to detect news topics from conversational topics and assessed their credibility based on various Twitter features [18]. The achieved a precision and recall of 70-80% using J48 decision tree classification algorithms. Canini et al. analyzed usage of automated ranking strategies to measure credibility of sources of information on Twitter for any given topic. They observed that content and network structure act as prominent features for effective credibility based ranking of users of Twitter [19]

III. RESEARCH METHODOLOGY A. Data Collection The data was collected from Twitter using the Streaming API. The Streaming API enabled us to extract tweets in real-time, based on query parameters like time of posting of tweet, number of characters in a tweet etc. Trends API from Twitter was used to obtain query terms. Tweets for trending topics were obtained, till the time the selected topics were in trend. Castillo et al. also used a similar framework to collect tweets using current trending topics (https://dev.twitter.com/docs/api/1/get/trends).

Table 1: Statistics of the Twitter Dataset Total tweets 5578 Tweets with URLs 1940 Number of Retweets 2263 Trending topics 576 Start date 10th September, 2016 End date 27th October, 2016

Our main hypothesis is that the level of credibility of information provided through social media can be analyzed. We believe that there are several characteristics that can be observed on Twitter itself, and that are useful to assess information credibility. These characteristics include:  The reactions that certain topics generate and the emotion conveyed by users discussing the topic: e.g. if they use opinion expressions that represent positive or negative sentiments about the topic;  The level of certainty of users propagating the information:  e.g. if they question the information that is given to them, or not;  The external sources cited: e.g. if they cite a specific URL with the information they are propagating, and if that source is a popular domain or not;  Characteristics of the users that propagate the information, e.g. the number of followers that each user has in the platform.

We propose a set of features to characterize each topic in my collections. These include some features specific to the Twitter platform, but most are quite generic and can be applied to other environments.

B. Annotation Scheme Human annotators helped to obtain the ground truth regarding the presence of credible information in tweets related to a news event. Human annotation for understanding the ground truth is a well-established research methodology[18].

A random sample of around 300 tweets per topic was picked up. Only tweets in English language were considered. Tweets by users in languages other than English were skipped by the annotators. Thus, non-English tweets were removed from the analysis.

The annotation scheme comprised of choosing one of the following options by the human annotators:  Definitely Credible  Seems Credible  Definitely Incredible  Seems Incredible  I can't Decide  Skip tweet The definition of credibility explained to the annotators was as follows: Oxford dictionary defines the term credibility as \the quality of being trusted and believed in." In the context of this research, we aim to assess the credibility of the information in the content of a tweet (message) by a user on Twitter. A tweet is said to contain credible information about a news event, if you trust or believe that information in the tweet to be correct / true. [] Each of the selected events was given 2-5 lines of description from credible sources such as CNN, BBC, and The Guardian. Redundant options such as ‘Seems Credible’ and ‘Seems Incredible’ were removed from the analysis. The 300 tweets per event were annotated by three different annotators.

All rights reserved by www.grdjournals.com

99

An Analysis of Credibility of Content on Twitter (GRDJE/ Volume 2 / Issue 5 / 016)

C. Computing Cronbach Alpha Cronbach's alpha is a measure of internal consistency, that is, how closely related a set of items are as a group. It is considered to be a measure of scale reliability. A "high" value for alpha does not imply that the measure is unidimensional.[] For the annotated data, Cronbach Alpha score was computed. The overall Cronbach alpha value for inter-annotator agreement for the total annotated tweets was 0.721. Alpha > 0.7 implies a high agreement between annotators []. We rejected all tweets for which all three annotators gave different scores and considered only the majority score (as given by at least 2 annotators) for a tweet.

IV. ANALYSIS In this section, we analyze a given tweet related to an event, automatically determine which events are newsworthy and assign a certain label of credibility.

A. Types of Features The primary characteristics that define a tweet are the features of the tweet itself, and the characteristics of the user who posted the tweet on Twitter. Following table 2 lists the types of features that can be extracted from a tweet and the user. The following two categories are considered by us:  Tweet based features: The message posted by the user having 140 characters comprise of data (eg. hashtags, URLs) and metadata(eg. retweet). Text semantics are not considered in this analysis.  User based features: The characteristics of the user who posted the tweet. Various properties such as number of followers, verified status of the user, are part of this category.

Table 2: Tweet and User Based Features Considered for the Analysis Tweet based features User based features Number of unique characters, number of hashtags, number of words, length of the Number of followers, number of friends, is a verified tweet, number of positive sentiment words, number of negative sentiment words, account, registration age, has URL, screen name, tweet is a retweet, number of retweets, number of emoticons, number of special length if screen name, number of statuses, geo symbols, tweet is a reply, has URL, number of URLs, number of @ mentions location, coordinated of geo location

B. Feature Analysis The distribution of features values was analyzed. A best-feature selection process was done over the 764 cases categorized as newsworthy tweets. We used a best-first selection method which initially had an empty set of attributes and then searches forward.

The following features were selected, as listed in Table 3.

Table 3: Best Features Selected using Best-First Selection Max Min Mean Has URL

1

0

0.616

Average status count

53831

Average followers count

9345

171 4

346 841

Average Registration Age

1225

1

344

Average Friends Count

1420

0

470

Average Positive sentiments

1

0

0.313

Average Negative sentiments

1

0

0.306

Smile emoticon

0.20

0

0.012

Has location in profile

1

0

0.161

C. Regression Analysis A logistic regression analysis was performed on the features listed in Table 2 to identify the good predictors of credibility of tweets.

To perform regression analysis, all the tweets annotated as definitely credible were considered for the positive class (dependent variable = 1) and the not credible class were considered for the negative class (dependent variable=0). The following features having p-value<0.001 were obtained as strong indicators of credibility:  Number of unique characters in a tweet  Number of characters in a tweet  Has pronouns and sad/happy emoticons  Presence of swear words

All rights reserved by www.grdjournals.com

100

An Analysis of Credibility of Content on Twitter (GRDJE/ Volume 2 / Issue 5 / 016)

The number of unique characters in a tweet indicated greater credibility, possible because tweets with URLs, @mentions, hashtags, have more unique characters. Such tweets are more informative and thus more credible. Tweets containing high number of sad emoticons and low number of happy emoticons were found to be really good credibility predictors. The presence of swear words in a tweet indicates personal opinion of a user and is thus less credible for an event. This is due to the fact that tweets reporting news about an event are neutral in nature and do not consist of biased opinions. The HasURL feature showed a positive correlation with credibility. This is because URLs prove to be more informative about an event since they refer to pictures, videos about the event.

Regression analysis also identified significant features based on message and user of the tweet. Mostly all tweet based features were found to be significant in determining the credibility. User based feature such as number of followers proved to be a strong predictor of credibility. Thus, we conclude that both the tweet based and the user based features are prominent in determining the credibility of tweets.

V. CONCLUSIONS AND FUTURE WORK Twitter being a largely popular micro-blogging site has become a news and information sharing platform. During significant events, users log on to Twitter to find information about the events and also to post about the event. Many naive users can easily be misled by the plethora of incredible information posted by some users. Thus, it’s essential to provide tools that can assess the credibility of information posted online.

In this paper, we mined tweets from Twitter related to ten high-impact events and analyzed the credibility of information present in each of the tweets. The annotators found that only 15% of the total tweets contained awareness information that was credible. We performed logistic regression to identify the significant features which can in turn provide information about the credibility. Effective tweet based features were presence of swear words, pronouns, emoticons, number of unique characters in a tweet; and user based features were length of username and number of followers. It was observed that credible news was posted by users that had written a large number of tweets in the past, and the tweets having many retweets. We showed that both tweet based and user based features were prominent in providing credible information about tweets.

For future work, we wish to use larger datasets and analyze other factor that characterize a topic as credible. We would also like to develop some self-learning mechanism to overcome the limitation of using human annotators.

REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20]

E. Agichtein, C. Castillo, D. Donato, A. Gionis, andG. Mishne. Finding high-quality content in social media. In WSDM ’08: Proceedings of the international conference on Web search and web data mining, pages 183–194, New York, NY, USA, 2008. ACM. Alonso, Omar, Carson, Chad, Gerster, David, Ji, Xiang,and Nabar, Shubha. Detecting Uninteresting Content in Text Streams. In SIGIR Crowdsourcing for Search J. Chen, R. Nairn, L. Nelson, M. Bernstein, and E. Chi.Short and tweet: experiments on recommending content from information streams. CHI '10, pages 1185-1194, 2010.

C. R. W. Watch. Leap of faith: Using the internet despite the dangers.http://www.consumerwebwatch.org/pdfs/princeton.pdf, October 2005.

S. Chhabra, A. Aggarwal, F. Benevenuto, and P. Kumaraguru. Phi.sh/$ocial: the phishing landscape through short urls. CEAS 2011, pages 92-101, 2011.Evaluation Workshop, 2010.

T. J. Johnson, B. K. Kaye, S. L. Bichard, and W. J. Wong. Every blog has its day: Politically-interested internet users’ perceptions of blog credibility. Journal of Computer-Mediated Communication, 13(1), 2007.

A. Java, X. Song, T. Finin, and B. Tseng. Why we twitter: understanding microblogging usage and communities. In WebKDD/SNA-KDD ’07: Proceedings of the 9th WebKDD and 1st SNA-KDD 2007 workshop on Web mining and social network analysis, pages 56–65, New York, NY, USA,2007. ACM. Pear Analytics. Twitter study. http://www.pearanalytics.com/wp-content/uploads/2009/- 08/Twitter-Study-August-2009.pdf, August 2009.

M. Naaman, J. Boase, and C. H. Lai. Is it really about me?: message content in social awareness streams. In Proceedings of the 2010 ACM conference on Computer supported cooperative work, CSCW ’10, pages 189–192, New York, NY, USA, 2010. ACM. H. Kwak, C. Lee, H. Park, and S. Moon. What is twitter, a social network or a news media? In World Wide Web Conference. ACM Press, 2010.

V. Lampos, T. D. Bie, and N. Cristianini. Flu detector - tracking epidemics on twitter. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD 2010), pages 599–602, Barcelona, Spain, 2010. Springer, Springer.

J. Sankaranarayanan, H. Samet, B. E. Teitler, M. D. Lieberman, and J. Sperling. TwitterStand: news in tweets.In GIS ’09: Proceedings of the 17th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems, pages 42–51, New York, NY, USA,November 2009. ACM Press.

T. Sakaki, M. Okazaki, and Y. Matsuo. Earthquake shakes Twitter users: real-time event detection by social sensors. In Proceedings of the 19th international conference on World wide web, WWW ’10, pages 851–860, New York, NY, USA, April 2010. ACM. A. M. Popescu and M. Pennacchiotti. Detecting Controversial events from twitter. In Proceedings of the 19th ACM international conference on Information and knowledge management, CIKM ’10, pages 1873–1876, NewYork, NY, USA, 2010. ACM. M. Mathioudakis and N. Koudas. TwitterMonitor: trend detection over the twitter stream. In Proceedings of the 2010 international conference on Management of data, pages 1155–1158. ACM, 2010.

A. L. Hughes and L. Palen. Twitter adoption and use in mass convergence and emergency events. In ISCRAM Conference, May 2009.

K. Poulsen. Firsthand reports from california wildfires pour through twitter. October 2007.

C. Castillo, M. Mendoza, and B. Poblete. Information Credibility on Twitter. In WWW, pages 675{684, 2011.

K. R. Canini, B. Suh, and P. L. Pirolli. Finding credible information sources in social networks based on content and social structure. In SocialCom, 2011.

J. Ratkiewicz, M. Conover, M. Meiss, B. Gon_calves, S. Patil, A. Flammini, and F. Menczer. Truthy: mapping the spread of astroturf in microblog streams.

WWW '11.

All rights reserved by www.grdjournals.com

101