What Works in Student Retention? ATTRACT WP8 Sub Report
Ana Lucas Isabel Gonรงalves (Coord.) Anna-Kaarina Kairamo (ed.) April, 2012
Our University Partners:
Contents 1. Questionnaire Background and Aims ....................................................................................................... 3 1.1 How to use the questionnaire ................................................................................................................ 4 2. Questionnaire Outcomes ......................................................................................................................... 4 2.1. Section I – Background Information ...................................................................................................... 4 2.2. Section II – Retention and Degree-Completion Rates ........................................................................... 5 2.3. Section III – Factors Affecting Student Attrition at Your School ............................................................ 5 2.4. Section IV – On Campus Retention Practices ........................................................................................ 6 2.5. Practices that have the greatest positive impact on retention ............................................................. 9 2.6. Assessed Retention Practices .............................................................................................................. 10 2.6.1 Assessed factors affecting student attrition ...................................................................................... 10 2.6.2. Assessed retention practices on campus ......................................................................................... 11 3. Conclusions ............................................................................................................................................. 12
1
Questionnaire Background and Aims
What Works in Student Retention? is a USA instrument, owned by ACT Company, that has been used since 1980, with a total of 4 implementations since it’s first application. The questionnaire is directed to all American private and public, technical and community colleges, and aims to assess the perceptions and the specific causes of attrition and the factors that may affect retention in higher education. ATTACT WP8 Student retention working group decided to test the effectiveness of the instrument in European context in spring 2011. The application of the questionnaire by the WP8 group was authorized by the ACT Company for the 2010 questionnaire version. The original questionnaire has seven sections, I.
Section I: Background Information.
II.
Section II: Retention and Degree-Completion Rates.
III.
Section III: Factors Affecting Student Attrition at Your School (42 factors).
IV.
Section IV: On-Campus Retention Practices (94 factors).
V.
Section V: Items that have the greatest positive impact on retention at your school.
VI.
Section VI: Permission for a brief follow-up questionnaire or phone call.
VII.
Section VII: Comments.
In the WP8 application only the first five were used, since the remaining two did not apply to this specific application. Being a North American instrument the questionnaire had some contingencies in its direct application to the European context, such as different higher education systems, different school contexts and different student populations. However, the interest of ATTRACT WP8 group was to test how this tool helps for instance, in trying to find common retention practices and factors among the WP8 partners, take notice and be aware of new and different retention practices; and reflect on how many of the practices be implemented and with what outcomes. This report sums up the results of this exploratory study.
3
1.1
How to use the questionnaire
Using the What Works in Student Retention? questionnaire requires the authorization of the ACT Company. For the 2010 version the permission can be asked to one of the authors following e-mails: wes.habley@act.org, mike.valiga@act.org, randy.mcclanahan@act.org and kurt.burkum@act.org, or by the ACT formal channels, using this e-mail address: Publications@act.org
After being submitted the permission for use (by e-mail, telephone or in person) the data should be gathered. Sections I – Background Information and II – Retention and Degree-completion rates, are submitted to a descriptive data analysis, which results in a characterization of the respondent schools. The information can be helpful to understand how schools with distinct background perceive and intervene on student retention. Regarding Sections III and IV, all the items are treated as mean values; supplemental analysis can be conducted, such as screening the results by school type (public, private, etc.), or some inferential analysis if the sample has the right dimension and the investigator possesses the needed variables.
2
Questionnaire Outcomes
Within ATTRACT project the questionnaire was sent in spring of 2011 by e-mail to all of the 9 partners Grenoble INP; Instituto Superior Técnico [IST]; Uppsala University; Royal Institute of Technology [KTH]; Aalto University; Linköping University [LiU]; Trinity College of Dublin; Politecnico di Torino; Technische Universität Darmstadt - using the Google Docs tool. The questionnaire was also sent to the external partner, K.U.Leuven. The partners themselves chose the representative who would fill out the questionnaire. The questionnaire had a 100% response rate. In the following subsections a sum up the results of the questionnaire will be presented.
2.1.
Section I – Background Information
Seven universities out of nine had no specific person on campus who was responsible for the coordination of retention programs. The two partners, who had a person responsible, said that person was an Associate/Assistant Provost. Only three universities offer online instruction, on average 5% of the total of the credit hours.
4
2.2.
Section II – Retention and Degree-Completion Rates
Section II aims to identify retention and degree-completion rates. Since partners had different ways to calculate retention rates, and the aim of What Works in Student Retention – ATTRACT WP8 Report was not to compare these values, authors wish to refer to WP8 comparison framework for further information (the section of Completion Rates). Regarding having a specific goal for its student’s degree completion rate, the majority of the partners assumed having none, in the moment when the questionnaire was answered.
2.3.
Section III – Factors Affecting Student Attrition at Your School
Section III contained a list of 42 institutional characteristics or factors that could affect student attrition. The factors were measured in a scale of 5 to 1 points, being 5. Major Effect, 3. Moderate Effect and 1. Little of no Effect on attrition at your school. Tables 1 and 2 summarise factors that received the highest and the lowest scores from respondents. Table 1. Factors Affecting Student Attrition at your School with the highest score Factors that are affecting more Student Attrition at your School
Highest Values
Level of student preparation for college-level work
3,8
Level of student commitment to earning a degree
3,6
Level of student motivation to succeed
3,5
Student study skills
3,5
Availability of academic advisors
3,4
Student personal coping skills
3,4
Student employment opportunities
3,4
Quality of interaction between staff and students
3,4
Level of job demands on students
3,3
Quality of interaction between faculty and students
3,3
Among the factors that the respondents considered as affecting most the levels of attrition are the ‘Level of student preparation for college-level work’, the ‘Level of student commitment to earning a degree’ and simultaneously the ‘Level of student motivation to succeed’ and ‘Student study skills’.
5
Globally, the majority of the selected factors are intrinsic factors that the school cannot directly control. It’s interesting that among the 10 identified factors, only 4 can be regimented by the school boards: ‘Availability of academic advisors’, ‘Quality of interaction between staff and students’, ‘Level of job demands on students’ and ‘Quality of interaction between faculty and students’. Table 2. Factors Affecting Student Attrition at your School with the lowest score Factors that are affecting less Student Attrition at your School
Lowest Values
Ratio of loans to other forms of financial aid
2,1
Residence hall facilities
2,1
Cultural activities
2,1
Campus safety and security
1,9
Programs to support students’ transition to residence hall living
1,6
The factors that respondents identify as having a smallest impact on attrition are all related to the social, cultural and material way of living in university: loans, facilities, activities, security and transition support programs.
2.4.
Section IV – On Campus Retention Practices
This section contained a list of 94 items, related to a series of programs, services, curricular offerings, and interventions that may contribute to retention within partner’s schools, measured in a scale of 5 to 1 points, being 5. Major Contribution, 3. Moderate Contribution and 1. Little of no Contribution to retention. To the WP8 application only 88 of the items were listed, because some of the items were not applicable to the European reality. In the present analysis we only considered the practices used by 5 or more of the respondents. The following tables resume the main results:
6
Table 3. Most frequent "On Campus Retention Practices"
Total Score
Nr. Respondents
Mean
Faculty use of technology in teaching
33
10
3,3
Faculty use of technology in communicating with students
33
10
3,3
Training for faculty academic advisors
26
9
2,9
Career exploration workshops or courses
26
9
2,9
Individual career counseling
29
9
3,2
Instructional (teaching) techniques
28
9
3,1
Interdisciplinary courses
27
9
3,0
International students
31
9
3,4
Most frequent "On Campus Retention Practices"
Within the most frequent retention practices held by the WP8 respondents, the ‘Use of technology in teaching’ and the ‘Use of technology in communication with students’ seem to be practices that all of the respondents considered as having a moderate contribution to retention.
Table 4. "On Campus Retention Practices" with the highest mean Total Score
Nr. Respondents
Mean
Peer mentoring
31
7
4,4
Internships
20
5
4,0
Staff mentoring
12
3
4,0
Vocational aptitude assessment
4
1
4,0
Personality assessment
4
1
4,0
Workshops in money management
4
1
4,0
Degree guarantee program
4
1
4,0
Mathematics center/lab
31
8
3,9
Integration of advising with first-year transition programs
19
5
3,8
Faculty mentoring
19
5
3,8
"On Campus Retention Practices" with the highest mean
The practices with the highest mean held by the WP8 respondents, found the most significant were the ‘Peer mentoring’, the ‘Internships’ and ‘Staff mentoring’,
7
Table 5. "On Campus Retention Practices" with the highest mean within the most frequent "On Campus Retention Practices" with the highest mean within the most frequent
Total Score
Nr. Respondents
Mean
Peer mentoring
31
7
4,4
Mathematics center/lab
31
8
3,9
Remedial/developmental coursework (recommended)
26
7
3,7
Supplemental instruction
29
8
3,6
Tutoring
29
8
3,6
Increased number of academic advisors
21
6
3,5
Early warning system
21
6
3,5
International students
31
9
3,4
Online learning support
24
7
3,4
Center(s) that integrates academic advising with career/life planning
20
6
3,3
Foreign language center/lab
20
6
3,3
Faculty use of technology in teaching
33
10
3,3
Faculty use of technology in communicating with students
33
10
3,3
Individual career counseling
29
9
3,2
Advising interventions with selected student populations
19
6
3,2
Writing across the curriculum
19
6
3,2
Extended freshman orientation (non-credit)
22
7
3,1
Study skills course, program, or center
25
8
3,1
Assessing student performance
25
8
3,1
Instructional (teaching) techniques
28
9
3,1
The practices with the highest mean within the most frequent held by the WP8 respondents, found the most
significant
were
the
‘Peer
mentoring’,
the
‘Mathematics
center/lab’,
‘Remedial/developmental coursework’.
8
and
the
2.5.
Practices that have the greatest positive impact on retention
The following table sums up the practices mentioned by respondents with greatest positive impact on retention: Table 6. "On Campus Retention Practices" with the highest mean within the most frequent
Practices with greatest positive impact on retention
Individual Career Counseling Supplemental Instruction
Performance contracts for students in academic difficulty
Tutoring Peer mentoring Assessing student performance
Mathematics center/lab Tutoring
Online learning support Peer mentoring
Mathematics center/lab Online learning support Study skills course, program, or center
Supplemental instruction Performance contracts for students in academic difficulty Faculty mentoring
Peer mentoring
Advising interventions with selected student populations Academic advising center Supplemental instruction
9
2.6.
Assessed Retention Practices
After all the respondents indicated the factors and the practices that were affecting retention at their schools, it was asked also which of the factors and practices were actually assessed. This data is not required in the ACT questionnaire, however WP8 group considered relevant to identify among the factors and practices perceived as having a positive impact on retention the ones that were actually assessed. 7 universities responded to the questions, and the results are presented as followed.
2.6.1.
Assessed factors affecting student attrition
The following table shows the assessed factors that respondents recognize as affecting attrition at their universities. Table 7. Assessed Factors Affecting Student Attrition at your School Nr of Universities that recognize the factor was affecting attrition 5
Mean scores
Level of job demands on students
5
3,3
Availability of academic advisors
3
3,4
Level of student motivation to succeed
2
3,5
Level of student commitment to earning a degree
2
3,6
Quality of interaction between staff and students
1
3,8
Quality of interaction between faculty and students
1
3,4
Level of student preparation for college-level work Student study skills
1
3,8
1
3,5
Student personal coping skills
1
3,4
Assessed Factors Affecting Student Attrition Student employment opportunities
3,4
According to the respondents, two factors were assessed by majority of the universities:
Student employment opportunities
Level of job demands on students,
According to the respondents following factors were also assessed by a couple of universities:
Availability of academic advisors’
Level of student motivation to succeed
Level of student commitment to earning a degree
Quality of interaction between staff and students
Quality of interaction between faculty and students
10

Level of student preparation for college-level work

Student study skills

Student personal coping skills
Some respondents also indicated, that some factors are assessed but with some limitations, eg. on course or programme level (not a university general practice) or only few times in past (not regularly), or when assessed tend to not reflect a direct impact on retention.
2.6.2.
Assessed retention practices on campus
In the following table are listed all the retention practices that are assessed at least by one partner university according to the respondents. Table 8. Assessed on campus retention practices Nr of Universities that assessed the practices
Mean scores
Individual career counseling
4
3,2
Study skills course, program, or center
3
3,1
Writing across the curriculum
3
3,2
Mathematics center/lab Faculty use of technology in communicating with students Center(s) that integrates academic advising with career/life planning
3
3,9
Tutoring
3
Instructional (teaching) techniques
3
Peer mentoring
2
4,4
Supplemental instruction Advising interventions with selected student populations
2
3,6
Assessing student performance
2
3,1
Faculty use of technology in teaching
2
3,3
Remedial/developmental coursework
2
3,0
Online learning support
2
3,4
Foreign language center/lab
2
3,3
Programs International students
3,4
Female students
1 1
Library orientation, workshop, and/or course
1
2,2
Increased number of academic advisors
1
3,5
Extended freshman orientation (non-credit)
1
3,1
On Campus Retention Assessed Practices
3 3
2
3,3 3,3 3,6
3,2
3,0
According to the respondents, all of the universities assessed at least one retention practice; however there is no retention practice assessed by all of the universities.
11
Nevertheless, the first two practices in Table 8 are assessed by the majority of the universities – Individual career counselling, and Study skills course/program/center. The following practices are assessed by three of the universities - Writing across the curriculum, Mathematics center/lab, use of technology in communicating with students, Center(s) that integrates academic advising with career/life planning, Tutoring and Instructional (teaching) techniques, and the remaining practices are assessed only by one or two of the respondents. Several respondents marked some of the practices assessed as being limited. Related to ‘Peer Mentoring’ and ‘Supplemental Instruction’, one respondent said that students participating usually had good retention rates but this is not necessarily due to just the mentoring project, as it tends to attract the most skilled and active students. Related to ‘Mathematics center/lab’, ‘Faculty use of technology in teaching’, ‘Faculty use of technology in communicating with students’ the same respondent states that the initiatives are appreciated by students but on retention no significance has been detected. Related to ‘Remedial/developmental coursework’, ‘International students’ and ‘Center(s) that integrates academic advising with career/life planning’, the same respondent affirms that this factor can help individual students but has no big overall contribution.
3
Conclusions
3.1. Summary of the ATTRACT “What Works in Student Retention” results The analysis of the WP8 partner’s responses (Section III) indicated some differences among the retention rates, the northern and central European countries had higher retention and the southern st
countries lower retention rates between the 1 and 2
nd
year. The main reason might be differing
student selection criteria and progress rules of the universities. For further information please visit the WP8 comparative framework. In terms of student degree completion rates, the realities are slightly different. Besides two of the respondents that presented really high completion rates, the remaining respondent’s present an average value of 65% student completion rates. It seems that when it comes to completion the geographical factor loses significance. The majority of the identified factors affecting student attrition (Section III) are intrinsic factors that the university cannot directly control: among the 10 identified factors, only 4 can be regimented by the universities. The factors that the respondents identify as having a smallest impact on attrition are all related to the social, cultural and material way of living in university: loans, facilities, activities, security
12
and transition support programs. The reflection of this result is that there are many factors, which seem to be out of the universities direct influence. The common on campus retention practices (Section IV) mentioned by partners are ‘faculty use of technology’ and ‘faculty use of technology in communicating with students’. The practices with the greatest positive impact on retention include ‘peer mentoring’, ‘internships’, ‘staff mentoring’ and ‘vocational aptitude assessment’. The practices with the highest mean within the most frequent chosen by the partners, were ‘peer mentoring’, ‘mathematics center/lab’, ‘and ‘remedial/developmental coursework’. For benchmarking purposes, it’s also interesting to note some On Campus Retention Practices that some partners have also chosen among the list as having a positive impact on retention: ‘supplemental instruction’, ‘performance contracts for students in academic difficulty’, ‘assessing student performance’, ‘study skills course, program or center’, ‘tutoring’ and ‘academic advising center’. For further information please visit WP8 case studies. The question whether the practice is assessed or not may have been interpreted by the respondents differently, some general trends showed up: mainly three factors were assessed by the majority of the partners – ‘student employment opportunities’, ‘level of job demands on students’ and ‘availability of academic advisors’, although factors such as ‘level of student motivation to succeed and commitment to earning a degree’ are reported by a few partners as being assessed, as well as ‘quality of interaction between staff and students and between faculty and students’, ‘student study skills’, ‘student personal coping skills’, and ‘level of student preparation for college-level work’ . The clear need for improving assessment in these areas among partners led to the organization of a workshop specifically directed at “Knowing Students and Working with questionnaires” (Uppsala, May, 9 – 10, 2012).
3.2. Comparison with North American submission results The following sub-sections try to sum up the main similarities and differences between the ATTRACT WP8 and the What Works in Student Retention? 2010 North American submissions. The comparison is made with the public sector of 4-year programs college/universities. Please notice that the samples have a distinct dimension, the North American example is a national survey, and the WP8 is the application to 10 European universities.
3.2.1 Background information and Retention and Degree-Completion Rates The majority of the WP8 schools had no specific person responsible for the coordination of retention 1
programs, which is a low result comparing with the USA 2010 questionnaire , where in the public sector
1
For Public four year colleges report go to: http://www.act.org/research/policymakers/pdf/droptables/PublicFour-YrColleges.pdf
13
colleges/universities only 31% of the schools indicated there was no person responsible for such coordination. Only two ATTRACT universities offered online instruction, but even those in a very small percentage, around 5% of the total of the credit hours. Once again, the results of the USA questionnaire are a little higher, in the same category, 39% of public schools offer at least 5% of online instruction. Regarding the Retention rate (for the ATTRACT universities which provided an answer to the question st
nd
on retention) between the 1 and 2 years, an estimated mean value of 66% was calculated which was lower than in the USA public sector of 74%. However, ATTRACT partner university degree-completion rates seem to be higher, with 70% of students finishing their studies, compared to 50% in the USA public sector. Having asked partner universities if a specific degree-completion rate goal had been developed, only 20% of the ATTRACT universities claimed to have one, a similar number to the USA respondents, 23%.
3.2.2 Attrition Factors Comparing the top 10 attrition factors with the highest means between the ATTRACT responses and the USA responses, we can conclude that the ‘level of student preparation for college-level work’, ‘student study skills’, ‘level of student motivation to succeed’, ‘level of student commitment to earning a degree’, ‘level of job demands on students’ and ‘student personal coping skills’ got similar values, standing as the main attrition factors in both samples. The main difference among the ATTRACT and the USA groups is the level of importance that the North American schools attach to the financial, social and economic factors.
3.2.3. On campus retention practices Comparing the practices with the highest occurrence, we can observe that only 5 of them are shared by the USA and European contexts: ‘faculty use of technology in teaching’, ‘individual career counselling’, ‘faculty use of technology in communicating with students’, ‘instructional (teaching) techniques’ and ‘interdisciplinary courses’. By comparing the most frequent practices with the highest mean, we can observe that only 4 are shared by the WP8 group and the USA study: ‘mathematics center/lab’, ‘tutoring’, ‘advising interventions with selected student populations’ and ‘study skills course, program or center’. In the North American context, there are two common practices that may be beneficial when applied to a European context: the ‘freshman seminar/university 101 (credit)’ and the ‘integration of academic advising with first-year transition programs’.
14
3.4 Reflection on the usability of the tool ATTACT WP8 Student retention working group decided to test the usability of the instrument in European context in spring 2011. The study shows that the USA tool has to be tailored to the European context. This includes also the need to define academic terms used. The challenge is that definitions for different academic terms vary from country to country and are potentially lost in translation. However, there is a need to find methods and tools to compare the practices of different universities and even more important, to learn from each other on what works in student retention. In order to benchmark each other’s practices, systematically gathered background information is required. The USA based tool may be a good starting point to develop a more specialised method for European universities.
15