Print (”Good Luck!”): Measuring the Effect of Autogenerated Social Encouragement on Student Anxiety

Print (”Good Luck!”): Measuring the Effect of Autogenerated Social Encouragement on Student Anxiety Collin A. Blanchard Holly A. Buff Travis D. Cook Raquel E. Dottle Gideon B. Luck Alani L. Peters Virginia L. Pettit Isaak Matthew Ramirez Jessica E. Wininger Abilene Christian University Abilene, TX 79601, USA {cab13e, hab13a, tdc12c, red13a, gbl12a, alp13d, vxp12c, imr13a, jew13b}@acu.edu

Abstract Requesting and receiving messages of encouragement on social media has previously been shown to significantly reduce test anxiety for students. We present an empirical study to test whether autogenerated messages of encouragement on social media are as effective as those from real people. Our results both confirm and extend previous research by showing that social encouragement can lower anxiety, but knowingly receiving autogenerated encouragement severely diminishes this effect.

Author Keywords Anxiety; social support; autogenerated support; social media.

ACM Classification Keywords H.5.3 [Information Interfaces and Presentation]: Group and Organization Interfaces–Collaborative computing

Introduction Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored.

For all other uses, contact the Owner/Author.

CHI’18 Extended Abstracts, April 21–26, 2018, Montréal, QC, Canada. © 2018 Copyright is held by the owner/author(s).

ACM ISBN 978-1-4503-5621-3/18/04. http://dx.doi.org/10.1145/3170427.3180287

Test anxiety plagues many college students throughout their university careers [1, 7]. Otherwise competent students perform poorly on exams because of test-specific anxiety [4]. In recent years, different approaches to relieve this anxiety have been proposed and some have been found helpful [2, 6]. A method that has been shown to be particularly successful for high anxious students is writing

expressively about the exam immediately before taking it [6]. One relatively accessible method found to be helpful is for students to use social media to request encouragement before their exams. Unfortunately, some students found it uncomfortable to share their anxiety with their peers as they deemed it to be vulnerable [3].

As a solution to this complication, this paper explores using automated messages of encouragement sent via Twitter to encourage those taking the quiz. In order to determine if the automated quality of the message affects the stress reduction, half of the experiment participants are informed that the messages come from automated Twitter bots while the other half are not. Stress levels are measured before and after the encouragement and completion of the quiz.

We designed our experiment to answer the following research questions: • RQ1: Will it matter whether the messages are autogenerated or from people? • RQ2: How does the level of student anxiety relate to number of correct answers on the quiz? • RQ3: How do students feel about seeking support from bots or other people prior to a quiz?

Related Work Causes of test anxiety include both the test itself and the mental state of the taker. Asghari et al. found that students are pressured to succeed and therefore develop test anxiety by fearing a negative reaction to their work. They propose that test anxiety is comprised of two parts. The first is a cognitive component, which is thinking about the test situation and its effects. The second is a physiological component, which can involve tense muscles and shaking, among

other symptoms. Female students are much more likely to experience test anxiety, and much more severely [1].

Cognitive distortions, such as catastrophizing, which is where one leaps to and begins imaging the worst possible outcomes, can be completely debilitating. Putwain et al. investigated the effect of cognitive distortions on test taking and anxiety and distinguished between the cognitive, physiological-affective component and behavioral components of this condition. They focused on the cognitive component to determine whether cognitive distortions are a cause of this anxiety or not, and found them to be a mediator of the relationship between worry and bodily symptoms, and academic achievement [5].

Some intervention methods for test anxiety that have been tested include expressive writing and humor. Ramirez and Beilock studied the effects of expressive writing and unrelated writing on high school students taking a multiple choice math test over an academic year. They found that expressive writing enabled higher-anxiety students to perform at the level of lower-anxiety students, but lower-anxiety students showed no improvement from expressive writing [6].

Students with high test anxiety need some sort of coping mechanism in order to reduce the anxiety that they feel and boost their test performances. Berk and Nanda explained that humor provided a mechanism that reduces anxiety and increased test performances. Unfortunately, Berk and Nanda concluded that anxiety levels before their experiment were already very low, meaning there was not much room for the levels to drop even lower. Berk and Nanda concluded that there were already methods in place that caused the class to have low-anxiety levels. [2].

Previous research has shown that messages of encouragement on social media has significantly reduced test anxiety for students and allowed high-anxious students to perform at the same level as low-anxious students. Deloatch et al. found that messages of encouragement from others on Facebook were significantly more helpful than students engaging in an expressive writing task. These Facebook messages caused a 21% decrease in anxiety for high test anxious students, whereas the expressive writing task actually caused a 61% increase in anxiety for low test-anxious students [3].

Methodology

Figure 1: The flow of the procedure for both control and experimental groups.

To answer our research questions, we performed a mixedmethods single factor between-subjects controlled experiment, which was approved by the IRB committee at ACU. The participants in this study were allowed to take an extra credit quiz for their Human-Computer Interaction class, in which the higher they scored, the more extra credit points they would be awarded. The single factor in the experiment was the anxiety intervention of social encouragement from Twitter bots in which half of the participants were told that the encouragement came from bots while others believed them to be from humans. See Figure 2 for an example of the autogenerated encouragement.

Procedure Once participants opted-in, they created dedicated Twitter accounts and were followed by the bot accounts written by the team. The bot accounts appeared to be real accounts belonging to other students at the same university, complete with profile pictures and bios. One class period before the quiz, students tweeted a request for encouragement with a required hashtag. The bots then searched for that hashtag and replied with encouraging messages. Students

Figure 2: An example of an autogenerated encouraging tweet sent to participants.

were asked not to look at these messages until shortly before the quiz.

On the day of the quiz, students were assessed using the trait portion of the State Trait Anxiety Inventory (STAI) form to determine their baseline test anxiety. The experimental group were told that the encouragement they were about to receive was autogenerated by bots, while the control group was told the encouragement would be from humans. Both groups then reviewed the replies to their tweets and then took the state portion of the STAI form to measure their current level of test anxiety. Then both groups took the quiz.

Following the quiz, they took a short self-assessment survey. The students anxiety was again assessed after the quiz using the STAI and a short self-assessment survey.

See Figure 1 for a visual summary of the procedure.

Measures The entire procedure took place within the Canvas learning management system (LMS), which was already in use for the course in which this took place. Before and after the quiz, we assessed participant anxiety using the STAI. Be-

Figure 3: STAI results plotted for each participant in the control group with the state portion as the y-axis and the trait portion as the x-axis.

Figure 4: STAI results plotted for each participant in the experimental group with the state portion as the y-axis and the trait portion as the x-axis.

fore the quiz, students completed the STAI-Trait portion to determine their usual level of quiz anxiety. After receiving the social media encouragement, students completed the STAI-State portion to determine their current level of anxiety going into the quiz. Upon completion of the quiz, the participants took a short self-assessment survey which gauged how they felt the Twitter encouragement had affected them and their quiz performance. This survey asked the following questions: (1) Do you think encouragement received from Twitter was helpful? (2) How did you prepare for this quiz? (3) How did you feel about requesting encouragement from other students? (4) Would you request encouragement from social media again?

Results A total of 27 students participated in this study. Participants were randomly split into two equal groups and were recruited from a Human-Computer Interaction course from a small university in Texas. Demographically, 73% were male and 27% female, and nearly all were junior or senior students from a technology-related discipline. There was no significant difference between the mean scores for the control (8.48/10) and experimental (8.32/10) groups. Therefore, it appears that knowing the encouragement was autogenerated did not affect the participant’s quiz score. Initial analysis of the STAI results showed an overall decrease in anxiety for participants after reading the encouraging messages, with nearly double the decrease in the control group.

After reading the messages, the control group’s measured level of anxiety lessened by 13.39% and the experimental group by 7.7%. This shows that the control group, who did not know that the messages were autogenerated, were more encouraged by 5.56% in comparison to the experimental group. Therefore, knowing the message was autogenerated decreased the effectiveness of the encour-

agement received. We performed a two sample t-test for unequal variance on both STAI-trait and STAI-state portions of that assessment in order to test for sameness between groups. Our statistical analysis showed the two groups of participants had unequal initial trait anxiety (p < .1175) and unequal state anxiety (p < .4451) after the intervention. We discuss this threat to validity further under Limitations.

The post-quiz self-assessment also yielded some insight into student perception of the intervention. We analyzed 2,474 words in the four response fields to determine the most used words and tagged them as emotionally positive, neutral, or negative (see Figure 5). We also analyzed student responses to the self-assessment question #4, "Would you request encouragement from social media again?" and tagged responses as either affirmative, neutral, or negative. This data is shown in Figure 6. The control group and experimental group had a difference of one person when asked the question, "Would you request encouragement from social media again?" Therefore, knowing the encouraging tweet was sent by a Twitter bot had no effect on whether or not participants were willing to ask for encouragement again in the future. The test scores did not improve by a significant percentage between the two groups.

Roughly half of the participants from both groups said they would not participate again.

Discussion After the experiment was conducted, all recorded data was processed using ATLAS.ti. We created tags for the Post Quiz Self Assessment to evaluate indications that the request increased, decreased, or seem to provide no change in the participants anxiety. Additionally, we created tags for individual quiz scores, and tags for ranking the participantâ&#x20AC;&#x2122;s level of anxiety as determined by the STAI State and Trait assessment. Overall, 18 unique tags were created,

Figure 5: Most frequently-occurring words in the post-quiz self assessment. Irrelevant words like "a", "the", and "for" have been removed from the results. Green represents positive emotions, blue represents neutral, and red represents negative.

10 of these tags were the integer score values of the quiz.

Once the tags were selected, the documents were coded resulting in approximately 190 tagged items. This qualitative analysis helped to discover the following.

Figure 6: Results from post-quiz questionnaire question "Would you request encouragement from social media again?" for all (top), control (middle), and experimental (bottom) participants. Red is no, green is yes, yellow is maybe.

Our study compared the effectiveness of encouragement on social media from human and automated sources. Because the experimental group, which knew the messages were from Twitter bots, were encouraged half as much as the uninformed group, we have shown that getting an encouraging message from an autogenerated source is not as effective as getting a message from a human. However, the informed group did still have a significant drop in anxiety. This means that the Twitter bots were still effective at helping test anxiety, but not comparatively to getting messages from real users.

The net decrease in anxiety was less significant than expected, as other social media interventions have shown a 21% decrease in anxiety [3]. We attribute some of this to

the lack of personal relationship in the intervention, as well as unfamiliarity with Twitter. Even though the experimental group believed the intervention was coming from humans, the fact that they had little to no relationship with these people could have diminished the efficacy of the intervention, though it was still significantly more effective than those who knew the messages had been auto-generated. Twitter and Facebook differ in that users are more likely to personally know the people that they are communicating with, which is one of the major distinguishing factors between our study and that of Deloatch et al. [3].

However, there was no significant difference between the two groupâ&#x20AC;&#x2122;s quiz scores. We believe that this is because the quiz was relatively easy for the participants as they performed quite well. Because there is an abundance of research showing anxiety correlates to quiz scores [2, 1], we believe that it is probable that a more difficult quiz would lead to the more anxious control group having performed more poorly on the quiz. Berk and Nanda saw a similar issue in their experiment, because their participants had very low test anxiety before the quiz and after the quiz.

Limitations There are several threats to validity to consider. The first is the small sample size (n=27). This small sample consisted primarily of the same major, Computer Science. Second, the quiz that we used to test anxiety was only for extra credit, which may have led to reduced pre-quiz anxiety.

The similarity between the quiz scores of the two groups suggests that our quiz may have been too easy for the participants. Third, the random assignment of test groups did not give us groups with verifiable sameness in the trait test.

Fourth, many of the students were unfamiliar with Twitter, or any social media platform, which may have affected the results. Twitter itself is also a more impersonal and brief plat-

form, as compared with other social media platforms like Facebook where messages have no character limits and connections are more likely to be people the user knows in real life. Therefore, the impersonal nature of Twitter may have decreased the efficacy of encouragement received, whether it was from a bot or not. Finally, our messages did not undergo any external verification to guarantee that they were truly encouraging in nature. If the messages were not considered to be encouraging to the participants, this would have undermined the intent of the study.

Conclusion As shown in previous research, test anxiety is extensively present amongst college students. Previous research has also shown that students who read encouraging messages from their friends on social networks exhibit decreases in test anxiety and improvements in performance [3]. Our research specifically looked at the comparative value of autogenerated messages to those of real people. We found that the autogenerated messages were roughly half as effective as messages that were believed to come from real people, at reducing test anxiety, answering RQ1. We also found that encouraging messages which came from people that the students did not necessarily know personally could reduce test anxiety among students. Even amongst students who benefitted, we found that many were unlikely to ask for encouragement through social media again in the future. In fact, the percentage of people who would ask for encouragement again was almost unchanged by whether they knew the messages were autogenerated, forming an overall negative response to RQ3. For score improvement (RQ2), we found that although there was a significant differences in anxiety levels between the two groups, there was not a significance between their test scores. While we are encouraged by the reaffirmation of social media’s role in reducing test anxiety, auto-generated messages may not be a promising method of anxiety reduction for test takers.

REFERENCES 1. Arezou Asghari, R Kadir, Habibah Elias, and Maznah Baba. 2012. Test anxiety and its related concepts: a brief review. Education Science and Psychology 3, 22 (2012), 55–60.

2. R.A. Berk and Joy P. Nanda. 2006. A Randomized Trial of Humor Effects on Test Anxiety and Test Performance. HUMOR: International Journal of Humor Research 19, 4 (2006), 425–454.

3. Robert Deloatch, Brian P Bailey, Alex Kirlik, and Craig Zilles. 2017. I Need Your Encouragement!: Requesting Supportive Comments on Social Media Reduces Test Anxiety. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 736–747.

4. Dave Putwain and Anthony L Daly. 2014. Test anxiety prevalence and gender differences in a sample of English secondary school students. Educational Studies 40, 5 (2014), 554–570.

5. David William Putwain, Liz Connors, and Wendy Symes. 2010. Do cognitive distortions mediate the test anxiety–examination performance relationship?

Educational Psychology 30, 1 (2010), 11–26.

6. Gerardo Ramirez and Sian L Beilock. 2011. Writing about testing worries boosts exam performance in the classroom. science 331, 6014 (2011), 211–213.

7. Frank C. Richardson, Harold F. Jr. O’Neil, Richard D. Jr. Grant, and Wilson A. Judd. 1973. Development and Preliminary of an Automated Text Anxiety Reduction Program for a Computer-Based Learning Situation.

U.S. National Institute of Education Technical Report.

(January 1973). Technical Report No. 20.