Assessment in Higher STEM Education: the Now and the Future from the students’ perspective

Assessment in Higher STEM Education: the Now and the Future from the students’ perspective Sofia Antera Stockholm University, Board of European Students of Technology

sofiante9@gmail.com Rita Costa University of Lisbon, Board of European Students of Technology

ritaflscosta@gmail.com Vasiliki Kalfa Aristotle University of Thessaloniki, Board of European Students of Technology

vicky.kalfa@gmail.com Pedro Mendes University of Lisbon, Board of European Students of Technology

pmendes1994@gmail.com

Abstract. The purpose of this paper is to provide input regarding the students’ perspectives on the assessment methods used in Higher Science, Technology, Engineering, and Mathematics (STEM) Education. Are traditional methods still effective? What are the students' perspectives on the diverse evaluation methods in Higher Education? To answer these questions, the Educational Involvement Department of BEST (Board of European Students of Technology), a nonprofit, non-governmental, non-political and non-representative student organization, organises BEST Symposia on Education, BSE (former Events on Education - EoEs), which aim to convene Higher Education stakeholders and raise the students’ engagement in Higher STEM Education.

By performing a secondary data analysis of the students’ perspectives as they were expressed and recorded in EoE Gliwice [1] and EoE Chania [2] reports, the current study shows that laboratory settings are supportive for combining the three most preferred learning techniques: discussion groups, practicing by doing and teaching others/immediate use. Moreover, it was concluded that the assessment on every evaluation system should combine the students’ attitude in class and feedback from professors. Final exams no longer appeal to students and cannot reflect the knowledge and skill set obtained. Professors, universities adfa, p. 1, 2011.

© Springer-Verlag Berlin Heidelberg 2011

and particularly educational policymakers should consider the students’ needs both when formulating a fair assessment system and creating/updating academic curricula.

Keywords: Assessment, evaluation methods, student perspective, BEST.

1

Introduction

Learning assessment can significantly vary in the way it is perceived, the way it is addressed and surely in the way it is implemented, especially considering the different stakeholders involved in the process. What professors consider best practice students reject as invalid or non-reliable. The era in which students are awarded credit hours for attending classes and the transmission of knowledge is realised through lecturing is sadly not over [3]. With learning perceived as an act, it is “no longer seen as simply a matter of information transfer but rather as a process of dynamic participation, in which students cultivate new ways of thinking and doing, through active discovery, discussion, experimentation” [1].

Participating in this system, all students receive the same materials and teaching regardless of their cultural, social and knowledge background. Those requiring more time to learn fail, having no opportunity to follow at their own pace. This underestimation of the learning individuality leads many to believe that students’ learning in tertiary education has not gained enough attention and has resulted in being one of the least sophisticated aspects of the Higher Education learning process [4].

In this paper, we briefly review the concept of validity of education assessment with a focus on fairness, in order to theoretically explain the students’ views on STEM education assessment as expressed during the Events on Education of Gliwice and Chania.

2

Learning and Assessment

By approaching learning as a constructive act of the learner [5], we can assume that assessment is the process of evaluating this act. Despite the interactive and dynamic nature of learning and hence the difficulty to validly measure it, the academic community tends to evaluate learning, process, and outcomes for several purposes, such as ranking students or improving the learning experience. As Hattie [6] highlights, in Higher Education ‘we implicitly trust our academics to know what they value in their subjects, to set examinations and assignments, to mark reliably and validly, and then to record these marks and the students move on, upwards, and/or out’.

In this line of thought, several assessment methods are implemented by tertiary education institutions representing different approaches to both learning and assessment and serving various purposes. Therefore, in education, assessment refers to a variety of methods and tools applied to measure, document and evaluate the academic readiness, the progress of learning, the competence acquisition and the

students’ educational needs 1. Nowadays, the efficiency of these methods is highly doubted not only by the scientific community but also by the receivers of these practices, the learners.

Investigating students’ general perceptions about assessment [7], aspects of perceptions emerging refer to the validity of assessment methods and to the concept of fairness, as perceived by students. Drew’s study shows that learners value selfmanagement and examinations are perceived as less supportive for the development of this competence, whereas deadlines per se are not considered as unhelpful. Deadlines increase self-discipline as well as the ability to study under pressure, and assessment can be seen as a powerful motivation to learn. In addition, clarity of professors’ expectations and assessment criteria are highly valued by the learners. Effective communication between the learners and the teachers is also appreciated. Finally, students tend to link feedback with support, since the latter is critical to boost selfconfidence [7].

2.1

Validity and fairness

Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationale support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessments [8].

According to this definition, the aim of assessment is to provide a comprehensive package of evidence that learning takes place, including the competences in practice and the degree they are developed [9].

Curricular validity ensures that the goals and objectives set by the curriculum agree with what students need to know. Curricular validity is a prerequisite for quality assessment since it examines the link between the learning objectives for the course and the desirable outcomes [10].

Construct validity refers to the relation between the assessment content and method and the learning objectives and is prone to dispute. Most assessments require general skills besides the subject domain (e.g. the ability to construct an essay). These competences can be assessed along with the subject-related ones or they can be considered implicit criteria in the learning objectives. Establishing construct validity, the professors need to ensure that the assessment content corresponds to the course learning objectives [10]. In the present study findings suggest that students doubt the construct validity of the assessment methods under discussion.

Finally, according to predictive validity, predictions made based on the assessment results are valid. This validity type is useful for selection purposes because it ensures that the current student performance is closely related to a future one [10].

Validity is related to the concept of fairness [5], considered a fundamental aspect of assessment, by students. Embracing more than cheating, the notion of fairness relates closely to the one of validity. Therefore, it is common that students express the view that a method may or may not be an accurate measure of learning.

Dimensions of fairness related to assessment methods included the control the students have over the examination process. Evaluation methods that require no active participation from the side of students are regarded as less fair. Moreover, if the as1

Edglossary.org.: https://www.edglossary.org/assessment/

sessment results are highly influenced by the factor of luck, the method is considered of low fairness. For example, when only one part of the course is examined, the factor of luck may lead to either good or bad results that do not reflect the actual learning [11].

Students tend to associate fairness with the time and effort invested in what they regarded as meaningful learning. Consequently, assessment methods which demonstrate this effort are perceived as fairer. Other dimensions of fairness include openness and clarity, meaning that methods which support better communication between the examiner and the examinee and allow feedback are seen as fairer [11].

3

Methodology

Reshaping education by offering students the opportunity to have a core role in its formation is the way towards quality education [12]. In this context, BEST Symposia on Education - BSEs (former Events on Education - EoEs) convene Higher Education stakeholders, with the purpose to strengthen students’ involvement in several aspects of Higher STEM Education, through exchanging views and practices with academics, industry representatives, and other education experts. BSE’s outcomes encompass interesting students’ perceptions on education and, hence, their further exploration is considered helpful for making the students’ voice heard.

To achieve a more objective diversity of students, more than 20 STEM students participated in each event, from different countries, educational and cultural backgrounds. The selection is based on gender, academic qualifications and origin to ensure diversity. In EoE Gliwice and EoE Chania, gender balanced is reported, while 17 countries are represented combining both events. Regarding the students’ academic qualifications, more participants have already acquired a bachelor’s degree, but some are still undergraduate, and others already have a master’s degree (more details on Annex A).

Based on the data from the EoE Gliwice [1] and EoE Chania [2] reports, a secondary data analysis has been done and its results are presented in the next chapter.

This study adopts a qualitative research strategy, with aim to highlight the views of students regarding assessment methods. Qualitative research focuses on social processes assisting in demonstrating patterns emerging among participants’ perceptions [13].

The data collected and analysed refers to the outcomes as they were reported by the facilitators of the EoEs. The facilitators consisted of both the event organisers, meaning BEST members – engineer students of various disciplines – and university professors that delivered sessions and workshops 2. The group of authors performs qualitative content analysis, in an attempt to interpret the outcomes of the EoEs, identifying latent content [14]. Starting from the existing theory, the authors seek in the data same, similar or different issues in the assessment methods as expressed by students. The assessment methods discussed during both EoEs were collected and the most interesting ones were selected to be present in this study. The criterium was the amount of information available in the reports expressing the participants’ viewpoints.

2

Find more details about facilitators in references [1] and [2].

This study is limited to assessment methods discussed during the EoEs, as well as the data are analysed based on the reports and some theoretical approaches as mentioned above.

4

Results

While collecting students' views on evaluation methods, a distinction between traditional and alternative methods was made. The goal of this separation is to differentiate regular evaluation methods from the ones that are occasionally used in Higher STEM Education. This perception is also shown through students’ opinions.

To facilitate the input gathering, students discussed the methods' advantages and disadvantages separately and proposed improvement points for evaluation systems in general.

Traditional methods.

Referring to traditional methods, oral, written and multiple-choice exams were discussed. Oral exams are found to enhance soft skills through practicing public speaking. This evaluation method promotes fairness since cheating gets harder. Oral exams are found to enhance soft skills through public speaking, which promotes fairness since it is harder to cheat, but at the same time may undermine it because of achieving a good grade which would correspond to their soft skills (for example presentation skills) and not actual knowledge on the subject. To increase objectivity in evaluation, students suggested the engagement of more professors as part of a jury, rather than a single examiner. On the other hand, written exams were mentioned to be objective and offer more chances to pass. However, students believe that they only cover a small part of the course syllabus and promote ‘learning by heart’, hence surface learning. To counteract these two points, students noticed the need for fullfledged course evaluation to encourage deep knowledge and raise fairness. As a third addition to the traditional methods, multiple choice tests were not considered to support actual learning either in knowledge, soft skills or creativity and only help surface learning [2].

Alternative methods.

Numerous alternative methods have been discussed, along with their advantages and disadvantages, as well as improving points.

Open book.

Open book is a method that creates positive feelings to students, since they do not have to memorise trivial information. It was also perceived as less stressful. However, the examination difficulty level was discussed as very low, meaning that complete and proper evaluation could not take place [2].

Projects.

Widely known to boost hard skills, projects were considered to promote the practical implementation of knowledge, according to students. Although projects were perceived as good evaluation methods, students raised some issues for further discussion.

Project development may be time consuming, especially considering that projects cover only part of the course. Meanwhile, among team members, evaluating students’ individual involvement is always a challenge. To counteract these issues, students proposed to divide bigger projects into smaller ones, to minimise the workload. At the end of the semester the smaller projects could lead up to a final presentation. Also, students acknowledge that there should be a way to evaluate individual equal contribution to a project. Competition between teams may be a way to promote participation [2].

Case studies.

Regarding alternative methods, case studies seem to be increasingly popular to students. Among the advantages discussed, case studies promote analytical thinking by asking students to work on a real case scenario, strengthening teamwork and team building and encouraging self-directed learning, since students perform their own independent research. Moreover, 3 out of 6 teams stated that case studies support soft skills development. Nonetheless, students claimed that there is a constant repetition of topics throughout the years and there is usually only one acceptable solution, the one favouring a professor’s particular point of view. A yearly update of case studies is recommended to tackle the previously mentioned issue [2].

Presentations.

Following the previous concepts, students discussed presentations. 4 out of 6 teams claimed that presentations help to develop soft skills and especially, gain knowledge on public speaking. Even though presentations were signified as important, the fear of evaluating presentation skills and not knowledge was expressed. Students identified presentations as stressful and mentioned the influence, importance and impact of the impression that they will leave on the professor. To overcome the obstacle of public speaking, students suggested that universities should offer courses on presentation skills. Thus, evaluation would be based on knowledge of the topic instead [2].

Practical labs.

Moving to a different type of evaluation method, practical labs were explored. They were perceived as a method to promote deep learning through the application of theoretical knowledge in real working situation. On that account, students expressed the view that practical labs lead to an increased possibility of passing the final exam, since they facilitate better understanding of the subject. However, students traced a loophole in this method by indicating that it cannot evaluate theoretical knowledge.

The combination with other evaluation methods may help improve practical labs [2].

Online Exams.

Following the advancements in technology, online exams were also discussed as a possible evaluation method. Lack of time restriction and speed in grading were among the advantages presented. Additionally, 3 out of 6 student groups emphasised the benefits of flexibility on the examination timing. Nevertheless, technical problems during the procedure seem to be common and as half of the students pointed out, it is relatively easy to cheat during online exams [2].

Homework.

Even though some may consider it as a traditional method, homework at universities is ranked among the alternative methods. In this case, advantages balance the disadvantages. Students claimed that the main benefits are independence, flexibility in work scheduling and learning through exercising. Furthermore, during homework students can come up with more complex problem solutions, due to time flexibility.

Contradicting the benefits, students mentioned that this method is hard to ensure objective evaluation of learning, since it is based on the individual’s ability to do quality studying and it does not prevent cheating and copying the work of others. Having different homework with the same level of difficulty can be one of the solutions to this issue, as students proposed, as well as having a time restriction for completing homework assignments [2].

Improvement points in evaluation.

During the discussions, students stated additional ideas on evaluation methods, with aim to better evaluate both theoretical and practical learning.

Students remarked that evaluation should be monitoring the progress of their studies rather than depend on the final exam. Furthermore, they claimed that feedback from professors is better than judging in an evaluation system, to promote deep learning. In a unanimous understanding, students expressed the necessity of combining more than one evaluation method to obtain a successful outcome in learning and deviate from the traditional approach of final exams. Considering a different vantage point, they proposed numerous possibilities to objectify the grading of students, by detaching evaluation from a single person, namely one professor, and distributing the role in various parties [2].

Recommended ideas such as self-evaluation and peer-evaluation, appointment of a ‘jury’ consisting of professors experienced in different fields and definition of diverse evaluation criteria per professor, show that students contemplate the importance of passing from traditional evaluation methods towards alternative ones [2].

During a Time in Teams (TiT) activity 3 , students were given a scenario of an assessment system to reflect and propose improvement points [2]. One of the teams, facing outdated lectures with no recording, low level of freedom and flexibility and an insufficient share of projects and topic diversification, pointed out the necessity for a continuous assessment of the evaluation methods to enhance their quality. For this purpose, both students and professors should be involved, sharing their feedback before and after the classes. In the same line of thought, continuous professional development for professors is crucial, especially with a focus on the pedagogical strategies they use. More specifically, students mentioned that professors should remain informed about trends in evaluation approaches and regularly work to develop communication skills.

Correlating different assessment methods with learning techniques, namely discussion groups, practice by doing (projects) and teaching others/immediate use, the only evaluation method that could evaluate all three simultaneously was lab work according to students, while case studies, real life case studies provided by companies and in engineering competitions, thought to assess two out of three. With only one, there were problem solving discussions, summer camps, mentoring and internships [1].

Regarding discussion groups, monitoring students’ progress and evaluating discussion effectiveness through asking questions in an hourly basis were suggested.

During projects, the importance of acknowledging the share of contribution by students was highlighted. Competition among teams was also considered of value.

Finally, when it comes to teaching others, a system of points was mentioned, where points are awarded to mentors if their students successfully finish the tasks given.

Evaluation can take place among team members as well [1].

5

Conclusions

Reflecting on findings, it is agreed that the evaluation process should not reside exclusively on examination results, since they promote surface learning and are not favoured by students due to their low construct validity. Students express a strong concern about the objectivity of evaluation in general and the validity of the given evaluation method. Their fear is explained by the importance of the grades in progressing with studies. In an effort to seek for more fair evaluation methods, students suggest the engagement of multiple examiner/professors and demand evaluation methods that examine the course as a whole and not partially. In the same line of thought, students tend to believe that the wider variety of methods, the better, with online exams, presentations and case studies being the favourite assessment types among students. Finally, the proposal of breaking bigger projects into smaller units,

3

For the TiT sessions the students were divided in four teams, each one representing a hypothetical institution with a given number of problems (six, in this case). The students were responsible to present improvement points, following brainstorming and debating practices.

also serves the purpose of rising construct validity for students, ensuring that the content of the subject is thoroughly assessed.

Soft skills seem to have a quite controversial place when evaluation is under discussion. They are mentioned as competences practiced and developed in all assessment methods. However, although their practice and improvement are seen as positive, students perceived soft skills evaluation as negative, since they correlate it with poor evaluation of the course content.

Students’ opinion aligns with Drew’s conclusions with respect to the importance of feedback. Feedback is perceived a friendlier form of evaluation and refers in a big degree to the progress made while studying. Therefore, with feedback the focus moves from evaluating the outcome to assessing students’ effort and progress. According to Drew [7], focusing on assessing progress and effort enhances the feeling of fairness.

Continuous assessment of the evaluation methods and their effectiveness is rendered as necessary by students, with feedback constantly shared between students and professors. This closer relationship between the stakeholders engaged in the process of evaluation can offer a multifaceted approach in selecting and applying evaluation methods corresponding to the needs of both the students and the institutions. This step is essential in moving towards assessment that is not solely translated into grades, but it actually monitors students’ progress by supporting their learning.

6

References

1. Manasova, D. , Merlier, A., Guliaeva, A., Wippich, A., Trajkovikj, N. (2016). “Be on the right track with SMART, learning - change the education of tomorrow!”. BEST Event on Education 2016, July 21-31 2016, Gliwice, Poland. https://issuu.com/bestorg/docs/eoegliwice 2. Kloster Pedersen, L., Sobrino Verde, C., Churyło, K., Pasovic, D. (2017). “Refreshing Education: Update, Rethink, Grow”. BEST Event on Education 2017, July 12-21 2017, Chania, Greece. https://issuu.com/bestorg/docs/eoe_chania_2017 3. Johnston, H. (2011). Proficiency-based education. Retrieved from ERIC database.

(ED538827) 4. James, R. (2010). Academic standards and the assessment of student learning: Some current issues in Australian higher education, Tertiary Education and Management, 9(3), pp 187-198.

5. Struyven, Dochy & Janssens, (2002). “Students’ Perceptions about assessment in higher education: a review”. Joint Northumbria/ Earli SIG Assessment and Evaluation Conference: Learning communities and assessment cultures, University of Northumbria at Newcastle, Longhirst Campus, August 28-30 2002.

6. Hattie, J. (2009). The black box of tertiary assessment: An impending revolution. in L. H. Meyer, S. Davidson, H. Anderson, R. Fletcher, P. M. Johnston, & M. Rees (2009), Ter-

7.

8.

9.

10.

11.

12.

13.

14.

tiary assessment and higher education student outcomes: Policy, practice and research, pp 259–275, Wellington, NZ: Ako Aotearoa & Victoria University of Wellington.

Drew, S. (2001). Perceptions of what helps learn and develop in education. Teaching in higher education, 6(3), pp 309-331.

Messick, S. (1989). Validity. in R. L. Linn (1993), Educational measurement, 3, pp 13– 103, New York, NY: American Council on Education/ Macmillan.

Gyll, S. and Ragland, S. (2018). Improving the validity of objective assessment in higher education: Steps for building a best-in-class competency-based assessment program. The Journal of Competency-Based Education, 3:e01058.

McAlpine, M. (2002). Principles of assessment. Luton: CAA Centre, University of Luton.

Sambell, K., McDowell, L., & Brown, S. (1997). ‘But is it fair?’: an exploratory study of student perceptions of the consequential validity of assessment. Studies in Educational Evaluation, 23 (4), pp 349-371.

Sambell, K., McDowell, L. & Montgomery, C. (2012). Assessment for learning in higher education. Milton Park: Routledge, pp 10-11.

Bryman, A. (2012). Social research methods, 4, Oxford: Oxford University Press.

Adamson, B. and Morris, P. (2007). Comparing Curricula. in M. Bray, B. Adamson, and M. Mason. (2007), Comparative Education Research: Approaches and Methods, pp 263283, Hong Kong: The University of Hong Kong.

Annex A EoE Gliwice (9 male/9 female)

Country FYROM Hungary Italy Belgium Ukraine Russia France Greece Spain Croatia Bulgaria Poland Moldova Portugal Turkey Estonia Romania

1 2 1 1 1 4 1 1 4 1 1 0 0 0 0 0 0

EoE Chania (10 male /13 female) 0 1 1 1 1 3 0 2 2 1 0 5 2 1 1 1 1

Table 1. Number of participants per country of each of the Events on Education.

Qualifications Secondary education Bachelor’s degree Master’s degree

EoE Gliwice 4 12 2

EoE Chania 9 12 2

Table 2. Qualifications of the participants of each of the Events of Educations.