Quality Assurance Handbook
Photo by Russ Ward on Unsplash
2 Authors: Clelia Paraluppi, QA Officer and Nida van Leersum, Policy Officer Approval and autorisation: Arno Smets, Director ES Education, and ES Management Team Date: 1 January 2023 Foreword ..3 Quality culture ..4 Quality standards and indicators .......................................................................... 8 Quality Assurance cycles.................................................................................... 12 Fraud, Code of Conduct, complaints .................................................................. 20 Evaluation instruments 24 Conclusions ........................................................................................................ 26 Annex 1: Table for the operationalisation of the Quality Standards & Indicators 28 Annex 2: Example of an evaluation instrument: Post-survey ............................. 42 Annex 3: List of accredited courses 47 Contents
Foreword
This Quality Assurance Handbook describes all the Quality Assurance (QA) activities relating to the work of the Extension School (ES). It comprises a summary of all short and long-term QA activities, information on when they will take place, and our implementation strategy
Quality has always been a guiding principle of our work. Moreover, when on 1 January 2021 ES became part of the University Corporate Office, we used the opportunity offered by this organisational restructure to look more actively at areas of improvement, and made QA one of the core tasks of the new organization.
To date, ES has taken huge steps in furthering its quality agenda. Some key achievements are:
• setting-up a Quality Assurance Board (QA Board) that also comprises independent education professionals. The QA Board is responsible for the quality of examinations, final assessments and certificates, and for monitoring that QA processes are being followed;
• establishing Learning and Assessment Regulations;
• hiring a full-time QA Officer, which is a new position within ES. The QA Officer is responsible for all quality-related activities as well as to for monitoring that quality processes are being followed;
• defining Quality Standards and Indicators at Institutional, Portfolio and Course level;
• running the first ‘ES quality & working culture survey’ completed by ES staff. This gives insights into the current status of quality within our organisation and indicates areas of improvement for the coming years;
• holding a quarterly meeting with all ES staff entirely dedicated to quality. Staff jointly brainstormed about quality values and how we define and co-create an improved ES quality culture based on those shared values.
In addition, we finalised and implemented a range of crucial policy documents and regulations, ensuring our commitment to quality in all aspects of our work. However, this is just the beginning, as we are on a journey to continuously improve by reflecting on what we can do better.
Thus far, we have identified three main areas of improvement: (A) having a quality culture that permeates everything we do, (B) operationalising our defined and shared quality standards and indicators and, (C) having robust QA processes to embed our quality standards and to help us to effectively deliver our quality activities whilst identifying areas for improvement.
This handbook describes the above three areas and, wherever possible, it outlines who is responsible or accountable for certain activities and how these are included in our working and reporting cycles. The handbook serves as a reference and guide for ES management and staff. We are proud of what we have achieved; everything we do is driven by our underlying commitment to enhance the quality of campus and online education.
3
The Extension School Management Team
Quality culture
One of ES’ overarching goals is to have a strong quality culture; it is one of the Strategic Targets 2025 (see figure on the right). Additionally, one of our core tasks is to manage the QA of the ES portfolio.
The Harvard Business Review defines a ‘true quality culture’ as an environment in which employees not only follow quality guidelines but also consistently see others taking quality-focused actions, hear others talking about quality, and feel quality all around them. ES’ ultimate goal is to create a culture in which employees feel an intrinsic drive and ownership to further strengthen quality within our organization through their daily actions.
In June 2022, we carried out a pulse survey to gauge the quality and working culture at ES. This was the first time we asked ES staff for their input on these topics. The findings from the survey serve as a baseline for the existing quality culture in our organisation and will be used to improve on those aspects that need it. Following the advice of the QA Board, the survey will be repeated in two years.
The survey results were also shared with ES staff in the quarterly
meeting held in August 2022 that focussed on quality. This meeting also served to collect input on the values that staff find important for a quality culture. Using the collective feedback, ES defined a vision on quality based on four shared values: collaboration, commitment to continuous improvement, professionalism, and trust.
4
Photo by Natasya Chen on Unsplash
Vision on quality
The quality culture we embrace at ES is based on a strong collaborative spirit with the shared goal of delivering an excellent experience to all learners.
We can be trusted to use our knowledge and expertise to deliver an impactful experience because we care about our diverse community of learners and their lifelong learning development.
Our teams work proactively with a focus on solutions: this approach enables us to improve the quality of our products and services. As an organisation, we foster a culture of learning, both for our learners and for ourselves; in this way we continue to grow and improve.
“Committed to excellence, because we care”
Photo by Jonny Caspari on Unsplash
Stakeholders and roles
Having a good quality culture means that quality is in everyone’s remit. However, the final responsibility regarding the quality of education is shared amongst different persons, teams
STAKEHOLDER
Executive Director
Director of ES Education
Quality Assurance Officer
Manager Education Support
Manager Business Development
Policy Advisor
Quality Assurance Board Management Team
Education Management Team
Academic Portfolio Directors
Faculty Coordinators
Administrative Officer Course Team
Portfolio & Product Managers
Learning Developers
Learning Technology Team
Moderation Staff
Educational Data Analyst
Communication Manager
Administration Team
6
and Boards. Some of these roles are legally prescribed in the ES Management Regulations (ES Beheersreglement).
ROLE IN QUALITY
Quality of ES strategy, innovation and operations.
Quality of ES education.
ES QA processes (process manager).
Quality of educational support and QA cycles (process owner).
Quality of marketing, enrolment processes and reporting cycles (process owner).
Regulations and all ES policy documents.
Quality of examinations and final assessments and thus certificates.
Advisory role on quality processes and quality of educational portfolio.
QA of the education offered.
Quality of education. Advisory role on QA and portfolio development of ES education.
Quality of portfolio for each ES educational theme.
Course evaluations and improvement plans.
Reporting and annual improvement plans (process manager).
Quality of course content and operations. Course team includes lecturers, teaching assistants and course coordinator (if appointed).
Quality of portfolio development and alignment. Portfolio and product management and review.
Course design and didactic quality.
Quality of technology.
Quality of course delivery.
Platform and learners’ data for course evaluations.
Sharing best practices on continuing education and lifelong learning.
Quality of administrative processes and systems.
7
Quality standards and indicators
In 2022 for the first time, ES developed and implemented a set of Quality Standards & Indicators to successfully benchmark all its quality activities. Currently, in the Netherlands there are no agreed standards for continuing education at the national level. However, to align with campus education, ES builds upon the existing quality standards of the Accreditation Organisation of the Netherlands and Flanders (NVAO) and makes these fit for purpose for online education courses and programmes1.
The ES Quality Standards & Indicators apply to three levels: Institutional, Portfolio, and Course. This three-level approach allows us to better organize and execute our QA System to guarantee a closer alignment with the existing quality culture, activities, and policies in place at TU Delft. For every standard and indicator, there are key stakeholders responsible or accountable for meeting the indicators and, ultimately, the quality standard. A ‘responsible’ stakeholder is the one who carries out the work individually or with a team to complete a task.
1 The quality standards are informed by other internationally renowned standards e.g., Online Course Quality Review Rubric (State University of New York;) E-Excellence Benchmark (European Association of Distance Teaching Universities); Quality Matters.
An ‘accountable’ stakeholder is the ‘owner’ of the work that they must sign off (approve) and is ultimately responsible for meeting the indicator; there are no financial nor legal implications associated with this role, with the exception of members of the Management Team as prescribed in the ES Management Regulations (ES Beheersreglement). There can only be one accountable person specified for each task or deliverable, and at least one responsible person, although others can be delegated to assist in the work required.
A comprehensive description, with measurements for each indicator and responsible stakeholders, can be found in Annex 1: Table for the operationalisation of the Quality Standards & Indicators. The three tables in the next section summarise those in Annex 1, giving for each standard the responsible and accountable stakeholders at institutional, portfolio and course level. Annex 1 provides the full description of the measurements and the stakeholders for each of the indicators; these will be evaluated and any necessary changes will be updated in future versions of this handbook.
8
Photo by Diana Polekhina on Unsplash
Institutional quality standards
The Institutional quality standards encompass core elements such as ES’ institutional philosophy, policies, and impact strategies. They ensure that as an organisation, we are successfully embedded within the university and a key player in the larger ecosystem of continuing education and lifelong learning. The table shows the five institutional standards (all associated indicators are included in Annex 1).
STANDARD
Responsible Accountable PHILOSPHY
ES’ mission is to educate the world and enhance the quality of TU Delft campus and continuing education. There is a corresponding strategic plan to support its educational philosophy.
IMPLEMENTATION
ES’ Educational Philosophy is realised in an effective, inclusive and learner-centred manner that is supported by a high-quality culture.
• Management Team
• Executive Director
• Management Team
• Policy Advisor
• QA Officer
• Management Team
EVALUATION AND MONITORING
ES systematically evaluates whether the intended policy objectives relating to educational quality are achieved.
DEVELOPMENT
ES focusses on the continuous improvement of its education, innovation and research.
• Policy Advisor
• QA Officer
• QA Board
• Process Managers
• Administrative Officer
• Management Team
• Administrative Officer
• QA Officer
• Education Support Team
• Communication Manager
IMPACT
Continuing education impacts TU Delft and beyond.
• Management Team
• Executive Director
• Director ES Education
• Executive Director
• Director ES Education
• Executive Director
• Chair of QA Board
• Manager Education Support
• Manager
Business Development
• Executive Director
• Director ES Education
9
Portfolio quality standards
The Portfolio Quality Standards reflect our strategy to provide courses and programs on specific themes, which are based on the expertise of the University, can attract the right target audience, and have a beneficial impact on TU Delft’s education and beyond. Therefore, our educational portfolio themes are of market and societal relevance, and we deliver online education in different formats to respond to the individual needs of lifelong learners and professionals. There are four Portfolio Quality Standards: the summary below identifies the relevant actors.
STANDARD Responsible Accountable
PORTFOLIO STRATEGY
The ES’ portfolio strategy should encompass themes based on the expertise of the University and on learners’ demand.
EVALUATION AND MONITORING
The ES portfolio is systematically evaluated for educational coherency and relevance
DEVELOPMENT, ACCESS AND REACH
The ES portfolio is broad and serving a diverse target audience across the globe.
• Academic Portfolio Directors
• Manager Business Development
• Portfolio & Product Managers
• Manager Business Development
• Portfolio & Product Managers
• Academic Portfolio Directors
• Portfolio & Product Managers
• Learning Developers
• Marketers
• Academic Portfolio Directors
IMPACT
The ES portfolio impacts TU Delft and beyond
• Manager Business Development
• Manager Education Support
• Portfolio & Product Managers
• Faculty Coordinators
• Director ES Education
• Director ES Education
• Director ES Education
• Academic Portfolio Directors
• Executive Director
• Director ES Education
• Academic Portfolio Directors
10
Course quality standards
The Course Quality Standards highlight the pedagogical model used in our unique course design: the cutting-edge course curriculum, learning activities, and technologies that create an optimal learning experience whilst fulfilling the relevant learning objectives.
STANDARD RESPONSIBLE ACCOUNTABLE
COURSE DESIGN AND DEVELOPMENT
Courses are designed and developed to provide a high-quality, stimulating, and engaging learning experience.
COURSE CURRICULUM
Courses enable learners to integrate knowledge and professional skills in the fields of Science, Engineering, and Design.
ASSESSMENT
Courses have an adequate assessment system in place according to the type of course.
COURSE DELIVERY
Course delivery methods encourage learners’ engagement and active learning through consistent moderation and feedback.
MONITORING AND EVALUATION
Courses are systematically evaluated for their educational quality and relevance.
TECHNOLOGY
Technology adequately supports the courses’ learning outcomes.
• Responsible Lecturers
• Course Teams
• Learning Developers Coordinator
• Learning Developers
• Learning Technology Team
• Responsible Lecturers
• Course Teams
• Faculty Coordinators
• Learning Developers
• Course Teams
• QA Officer
• Policy Advisor
• Faculty Coordinators
• Learning Experience Coordinator
• Course Teams
• Course Moderators
• Learning Developers
• Policy Advisor
• Data Analyst
• Course Teams
• Learning Developers
• Portfolio & Product Managers
• QA Officer
• Data Analyst
• Course Teams
• Learning Developers
• Learning Technology Team
• Learning Experience Coordinator
• Policy Advisor
• Portfolio & Product Managers
• Manager Education Support
• Process Manager Open Education
• Responsible Lecturers
• Manager Business Development
• Manager Education Support
• Responsible Lecturers
• Director ES Education
• Chair QA Board
• Responsible Lecturers
• Manager Business Development
• Manager Education Support
• Learning Experience Coordinator
• Manager Education Support
• Chair QA Board
• Executive Director
• Manager Education Support
11
Quality Assurance cycles
Following the approach described in the TU Delft Education Quality Assurance Plan, ES also bases its QA System12 on the Plan–Do–Check–Act (PDCA) cycle as shown in the figure on this page. The system provides a clear and continuous QA process for the ES organisation, its portfolio, and the individual courses – with the only exception of the online academic courses, as they are already part of the QA process for campus education. Their evaluation, therefore, sits within the faculty offering them.
In practice, the documentation ES uses in applying the model may differ from the standardised one used by faculty as it regards the programs, delivery methods, learners and legal status of the particular education offered, which diverges from that of regular campus education.
We apply the PDCA cycle to each of our three defined levels (Institutional, Portfolio, Course) to ensure a better alignment of quality standards and indicators.
It means that ES can organise the QA cycles for each level more effectively, taking into consideration the involved stakeholders, specific deliverables, and relevant time frames.
The iteration of the steps in the cycle guarantees the continuous improvement of our institution, products and services; its successful execution also rests on having in place appropriate QA governance, including comprehensive descriptions of who is responsible for carrying out which tasks and following which timeline.
2 A PDCA circle or Deming circle is a set of four successive, mutually connected activities with which the quality of an object can be systematically improved.
12
Photo by Cats Coming on Pexels
CHECK
ACT PLAN
Continuous improvement
DO
Institutional QA cycle
Our QA governance at Institutional level is realized in consultation with the ES Management Team, Strategy Board, Educational Advisory Board, and QA Board. The main reports are shared with the University’s Executive Board (EB) and the faculties. The PDCA cycle is implemented as follows:
Define the ES Strategic Plan, Annual Planning and Yearly Budget – taking into consideration the ES Governance Structure and its Educational Philosophy –in order to accurately define ES’ objectives.
Ensure transparency of findings and evaluations by disseminating these through the ES quarterly and annual reports for the faculties and the EB, as well as via the annual evaluation and the 5-year institutional evaluation, as included in the ES QA and reporting cycle. Additionally, analyse and evaluate the impact of online education on campus education, so that recommendations can guide an improved integration of online offerings into campus education and beyond.
Collect highly aggregated learner and platform data, and revenue streams, to measure results against the ES’ objectives defined in the Strategic Plan. Data is also collected by surveying instructors on the impact of online courses on their campus teaching.
Wherever possible implement the recommendations for continuous improvement of ES’ education and services, which result from the ES evaluation processes: these will be included in a summary section in the ES QA Annual Report.
13
PLAN DO CHECK ACT Multi-year strategic plan Annual planning Budget Governance structure ES Philosophy Recomendations for Continuous Improvement Institutional evaluation Learning Platform Data Instructors surveys Data Learners surveys Data Annual reports Campus use and impact report Annual evaluation QA board annual report Quarterly Update and Reports Institutional QA Cycle PLAN DO ACT CHECK
Portfolio QA cycle
The educational content of the ES courses is guaranteed by faculty members; whereas the processes that facilitate education, including the QA process, are under the responsibility of ES. Therefore, the Portfolio QA Cycle can only be realized in close consultation between these two parties.
PLAN
Ensure the development of relevant strategies for the educational themes of the ES portfolio, including their alignment and coherency, and the existence of an appropriate management and governance structure to allow the realization of the strategy for each educational theme. The Director of ES Education is primarily accountable for the development of a portfolio of courses and programmes offered by ES that meets academic standards societal demand, market relevance, and the needs of external learners. The Academic Portfolio Directors (APDs) advise on the direction and opportunities within their portfolio theme in close collaboration with TU Delft faculties and the ES Portfolio & Product Managers (PPMs). The Faculty Coordinators are key in safeguarding alignment with faculty strategy, while the APDs guarantee portfolio alignment with the university-wide strategic themes.
DO CHECK ACT
Make recommendations regarding the ES portfolio offering and overall strategy based on the assessments and consultation on the abovementioned information and data analysis. The Portfolio Review will indicate which categories of courses require a more in-depth review and which should be prioritized. Strategic recommendations may address portfolio development, business and education strategies, and the availability of data and information management.
Conduct a yearly Portfolio Review covering the entire ES offering; this is executed by the PPMs who assess information and data about these five perspectives:
• pedagogical quality
• business performance
• course relevance for the portfolio theme
• relevance for faculty
• learner feedback.
Faculty Coordinators are crucial in providing data on the impact of courses for campus education and their relevance for faculty. In this process, Learning Developers provide input concerning the didactic aspects of all courses. The Portfolio & Marketing Team provide input on the business performance against industry benchmarks. APDs check whether courses align with the portfolio themes (fit with portfolio). The Educational Data Analysis Team provides learner feedback through the elaboration of aggregated survey results. Put into practice the recommendations from the Portfolio Review, including executing improvement plans for the revision and update of individual courses, or for (dis) continuing them. Strategy sessions are planned regularly to review and update the portfolio strategies for relevance and alignment; the ES Management Team will consider recommendations regarding policy and procedures as well as information management. Included in such sessions are also the recommendations from the analysis on the impact of online education on campus education to facilitate further integration between the two.
14
Portfolio Themes Strategy and Alignment
LLL Strategy Management structure
LLL Strategy Review
Campus Impact Report
Portfolio Strategy Report
Portfolio Review Report
Campus Impact Evaluation
Portfolio Review Recommendations
Portfolio Strategy Review
LLL Projects Evaluations
Market Research
Portfolio Review
Learners’ and Platform Data
15
DO
PLAN
ACT CHECK Portfolio QA Cycle
Course QA cycle
The Course QA Cycle comes to life through the systematic iteration of the four PDCA steps.
Course Improvement Plan
Course Development Roadmap Pedagogical Model
Accessibility and Usability Guideliness
Trainings and Course Creation Resources
Platform Data
Learners’ Survey Data
Course
Moderation Feedback Log
Assesment Results
Complaints Log
Course Evaluation Report/Survey
Course Evaluation Meeting
Feedback from Instructors and Learning Developers
16
PLAN DO ACT CHECK Course QA Cycle
Ensure that our approach to course and curriculum design and development is carried out across our entire portfolio using pertinent course development processes, training and resources. Our approach follows our unique pedagogical model, based on the Online Learning Experience Principles, a learnercentred approach, and open practices. Learning Developers are responsible for providing Course Teams with the necessary guidance to adhere to the Quality Standards by following quality control checks throughout the course creation process. Staff need to have access to resources and be able to complete the necessary training before they can start any moderation activity as part of the course delivery, and in compliance with the relevant quality standards. Moderators monitor activity in the courses and resolve or escalate any issues that may arise and hinder learners from completing the course.
PLAN CHECK
Conduct a course evaluation at the conclusion of every 1st and 2nd course runs for continuous improvement. After the first two runs, courses will be evaluated at regular intervals of three runs (see figure on the next page) taking into considerations the five perspectives identified in the Portfolio Review.
Participants involved in the evaluations are:
• Course Team
• Faculty Coordinator
• Marketer, Learning Developer, Portfolio & Product Manager.
The evaluations take the following input into account:
• course evaluation report (for MOOCs 1st runs and Programmes) or learner survey results for 2nd runs, ProfEds, and further runs
• moderation feedback
• learning developer feedback
• instructor feedback.
Part E contains a complete list of these evaluation instruments.
Utilise a whole range of learner data gathered through the completion of pre and post surveys, aggregated platform data and activity, course assessment results, and learner feedback. The Data Analyst in the Learning Technology Team collects and analyses data to inform the full evaluation that takes place once the course is concluded.
Produce and share a course improvement plan that includes considerations based on the evaluation. The improvement plan may refer, as necessary, to the:
• didactic quality of the course
• value proposition/About Page, promotional activities
• course delivery methods (instructor-paced vs self-paced, moderation by Learning Experience Team, trainings necessary for TA)
• list of actions for improving the course before the next run.
17
DO
ACT
Evaluation cycle of individual courses
Evaluations will be planned at regular intervals of three runs. This will allow to systematically monitor the course quality and learners’ satisfaction.
Course evaluation process
The teams in the figure below are involved in the various phases of the evaluation process and are responsible for defining and implementing the course improvements.
18 Evaluation team Course team Share evaluation report or survey results Collect learner and platform data Provide moderation feedback Course Team, Learning Developer, Product & Portfolio Manager, Marketer, Faculty Coordinator Define necessary improvements Evaluate course Make adaptations Course team Course starts
course evaluation course evaluation course evaluation course evaluation evaluation upon request evaluation upon request evaluation upon request evaluation upon request Run 1 Run 2 Run 5 Run 6 Run 7 Run 8 Regular course evaluation frequency Evaluation in exceptional cases only Run 3 Run 4
Reporting cycles and structure
The table below provides an overview of all reporting activities organized by timeline, divided by type of activity and expected reporting material, and by the responsible teams or Boards. Wherever possible, ES staff will have access to transparent and timely reporting materials.
Activity Report Responsibilities
ES Strategic Plan divided in:
ES Multi-Year Plan
(MYP) - every 6 years
ES Management Team
ES Strategy Board
Multi year
Quality Standards & Indicators review
Annual roadshows
Annual planning
Annual evaluation
Mid-term updateevery 3 years
ES institutional evaluation – every 5 years
ES Management Team
QA Officer
ES Management Team
ES Management Team
ES Management Team
ES Annual Report
ES Annual Faculty Report
ES Management Team
ES Management Team
Annual
ES Strategy Board Meeting
Quarterly update
ES QA Board Annual Report
ES Education Advisory Board Portfolio Review report
ES Education Advisory Board Portfolio Strategy report
Quality Assurance Board
Education Advisory Board
Education Advisory Board
ES Strategy Board
ES Executive Director
ES Quarterly Report to faculties
Quarterly
Education Advisory Board meeting (bi-annual)
QA Board meeting
ES Management Team
Education Advisory Board
QA Board
19
Fraud, Code of Conduct, complaints
To ensure learners have a good learning experience, the ES Learning and Assessment Regulations (LEARN) have been put into place, and we expect all learners to comply with the TU Delft Code of Conduct. An important part of these regulations concerns fraud and how to deal with it when it occurs.
According to the ES LEARN, fraud is committed when it is not possible to accurately assess the knowledge of learners. Fraud includes the use of forbidden materials during assessments, copies of the assessment questions or assignments beforehand, or sharing assessment questions with others. It also includes committing plagiarism (such as copying the work of others and presenting it as own work, either on purpose or due to carelessness and/or incomplete referencing).
In cases of fraud and/or plagiarism, the necessary steps to be taken, and who are the stakeholders involved, varies. The process for each type of course is explained below. Note that there is no ES process in the case of Online Academic Courses as these are a faculty responsibility: if fraud is suspected or detected during the final assessment, the Board of Examiners of the faculty offering the course can decide on the applicable sanctions. In such cases, the responsible lecturer should reach out to the relevant Secretary of the Board of Examiners.
20
Photo by Tim Gouw on Unsplash
Fraud and code of conduct in MOOCs
The main lecturer is responsible for dealing with individual cases. In case of large-scale fraud (relative to the number of course participants) or a breach of the TU Delft Code of Conduct, Course Teams should contact the ES QA Board via email (QABoard.ES@tudelft.nl). The Board can advise the Course Team to allow a re-take of the assessment/assignment, or to stop/revoke the issuance of the course certificate. The QA Board will react in writing within five days from the time the Course Team has reached out for advice. Should the Course Team be unsure whether they face a case of large-scale fraud, they should reach out to the QA Board for advice. The figure below describes the full process.
21
Individual learner commits fraud in MOOC Course Team is consulted Re-take is allowed Learner is informed and certificate issuance is stopped or revoked Learner is informed and may do a retake Learner completes the course Learner does not complete the course Large-scale fraud is commited MOOC Course Team is informed QA Board advises Course Team Yes No
Fraud and Code of Conduct in ProfEd courses and programmes
The Course Team/lecturer should contact the ES QA Board via email (QABoard. ES@tudelft.nl) regarding cases of fraud or breach in the Code of Conduct in ProfEd courses as indicated in the Learning and Assessment Regulations
The Board can decide on the applicable sanctions (depending on the seriousness of the case) and will communicate its decision within five working days to the Course Team and the learner(s).
22 Fraud or breach of Code of Conduct is detected in ProfEd Course Team is consulted Decision is taken that results are invalid and a retake is allowed Learner is informed and certificate issuance is withdrawn or revoked Learner(s) informed and allowed a retake Course is completed Course is not completed QA Board advises Course Team Yes No
Handling complaints
The complaints procedure is an important QA aspect. It helps to identify the elements that can hamper the quality of education. ES follows an internal complaints process when dealing with complaints – this is described in the figure at the bottom of the page.
All learners are made aware of the TU Delft Terms & Conditions for Contract Education at the time of enrolment. Article 13 of these Terms & Conditions gives information on the procedure that applies if learners wish to express a complaint about one of our products, the ES organization, or a ES staff member. The complaint should be submitted as soon as possible, and no later than four weeks after the completion of the course (Article 13.1). Learners must do so by stating in writing the reasons for their complaint, providing additional information such as the name and address of the complainant, description of the conduct about which the complaint is being made, and the name of the related course (Article 13.2).
The Administration Team receives complaints from learners enrolled in ProfEd courses via the functional mailbox or via the dedicated feedback form on the TU Delft Online Learning website. The Administration Team determines which department should handle the complaint or escalates to the relevant Course Team. The Administration Team also advises the complainant about the expected response time.
The edX Support Team receives complaints from learners enrolled in a MOOC via the dedicated online form available on third party platforms. Depending on the nature of the complaint, the Learning Experience Team in charge of moderation will either directly address the complaint or escalate to the Course Team. They will also record the frequency and type of complaints in a feedback form; such information will be used to inform future course evaluation. Additionally, Learning Developers will record complaints that required an escalation to the Course Team in a dedicated repository, which is shared with the QA Officer.
In case the learner is not satisfied with the proposed solution, they may contact the Central Complaints Desk, which will escalate their complaint to the ES Executive Director.
Should the nature of the complaint be a dispute on assessment results, Course Teams unable to resolve this can escalate the case to the ES QA Board for a binding settlement.
On a quarterly basis, an overview of all complaints is shared with the ES Director of Education and the QA Board, who provide advice on the actions to take (if any) to improve ES products and services.
23 Learner(s) expresses complaint ES staff receives complaint Complaint can be resolved by staff Complaint is escalated to Course Team and the complaint is resolved Decision is communicated to learner Complaint is closed, case is recorded in complaints repository Complaint is closed, case is recorded in complaints repository Yes No Learner escalates complaint to Central Complaint Desk Central Complaint Desk escalates to ES Executive Director and complaint is resolved No No Yes
Evaluation instruments
Several instruments are devised and used to measure to what extent our quality criteria are met; for example these could be surveys or analysis of platform data on learners’ completion rates and their level of interaction within the course. The tables below describe the instruments currently in use.
Enrolments analysis
What Information on number, type of learners (verified vs audit) and demographics. Information is based on platform data
Target group Learners
Function Provides information on the target audience
When At the conclusion of every course run
Feedback
Shared and discussed with Course Teams during course evaluation for 1st runs.
For following runs information is made available to Course Teams
Pass rate analysis
What Analysis of assignment attempts and pass rate of the course. Information is based on platform data
Target group Learners
Function Provides information on the subject level and student learning
When At the conclusion of every course run
Feedback Shared with Course Teams during course evaluation
Course engagement analysis
What Analysis of learners’ activity and engagement with the course (material). Information is based on platform data.
Target group Learners
Function Provides information on the type and frequency of the learners’ engagement with the course material
When At the conclusion of every course
Feedback Shared with Course Teams during course evaluation
24
Photo by Ali Kazal on Unsplash
Pre-survey
What Survey received by all learners
Target group Learners
Function Provides learner demographics information, educational and professional background, motivation for studying
When At the start of each course
Feedback
Post-survey
Shared with Course Teams during course evaluation. For subsequent runs information is made available to Course Teams
What Survey received by all learners
Target group Learners
Function Provides information on the general quality of and satisfaction with the course
When At the end of each course
Feedback
Shared with Course Teams during course evaluation for 1st runs. For subsequent runs information is made available to Course Teams
Quality and workplace culture survey
What Survey about the perceived quality culture within the organization and workplace satisfaction
Target group
ES staff and affiliated members of the ES (Academic Portfolio Directors, Faculty Coordinators)
Function Provides information on the organisational culture with regards to quality and the degree of work satisfaction that ES employees experience
When Every 2 years
Shared with ES Management Team and presented at ES quarterly meetings
Feedback
Campus impact survey
What Survey on the reuse and impact of online education and material into campus education
Target group Lecturers
Function Provides information on the extent and impact that online courses have on campus education and beyond
When Every 2 years
Shared with whole ES
Feedback
25
Conclusions
This first edition of the ES QA Handbook represents a significant step towards documenting an improved QA system. In future, we will carry out concrete checks and plan evaluations to measure the efficacy of the described PDCA cycle at all levels. Given the very recent publication of the ES Quality Standards & Indicators, and implementation of the QA cycle at Institutional and Portfolio level, adjustments are to be expected.
In a similar manner, we expect updates and amendments to this handbook and its annexes to be made after one year, including a review of the measurements and stakeholders defined in the operationalisation of the standards (Annex 1). Changes will result from ongoing consultations with several stakeholders and from their feedback on the implementation of the standards and quality cycles, and including the handling of complaints and fraud cases.
26
Photo by Artur Aldyrkhanov on Unsplash
After the first year, the QA Handbook will be updated every three years and be, wherever possible, aligned to the TU Delft Education Quality Handbook (Onderwijskwaliteit Handboek).
It is the intention to keep the conversation around the quality of education ongoing and place this topic at the centre of our activities. Therefore, anticipating changes to the way we plan, organize and execute our QA is a positive sign that we aim to continuously improve our activities and the quality of education and services we offer our learners.
27
We are committed to excellence, because we care.
Annex 1: Table for the operationalisation of the Quality Standards & Indicators
Institutional Standards
PHILOSOPHY
ES mission is to educate the world and enhance the quality of TU Delft campus and continuing education. There is a corresponding strategic plan to support its educational philosophy.
Standard Measurements
Responsible Accountable
1.1
ES EDUCATIONAL PHILOSOPHY IS IN PLACE AND ALIGNED WITH THE PHILOSOPHY OF TU DELFT
ES Educational Philosophy defined and communicated.
Yearly review as part of QA cycle
Manager Education Support Director ES Education
1.2
ES STRATEGIC PLAN IS IN PLACE TO ACHIEVE ITS MISSION AND EDUCATIONAL PHILOSOPHY
ES Yearly Strategic Plan. Strategic targets and timelines are shared with ES staff including at a dedicated Quarterly meeting. The input is used to inform the following year Strategic Plan.
1.3
OPEN EDUCATION IS A CORE ASPECT OF THE ES PHILOSOPHY
Included in the ES Educational Philosophy.
ES follows TU Delft OER Policy. OCW policy. OCW publication process.
Management Team Executive Director
Management Team Executive Director
1.4
ES COMMUNICATES ITS VISION AND STRATEGIC PLAN TO ALL STAKEHOLDERS
Strategic targets and timelines are shared with ES staff during a dedicated quarterly meeting. The input is used to inform the following year Strategic Plan.
1.5
ES CLEAR GOVERNANCE STRUCTURE UNDERPINS ITS PHILOSOPHY
1.6 ES HAS THE REQUIRED RESOURCES TO SUPPORT ITS OPERATIONS.
Internal Audit Evaluation for adequate governance structure. Evaluation and feedback on whether ES governance really facilitates the achievement of its goals.
Yearly budget.
Annual Planning including personnel and systems requirements.
Invite HR advisor to MT meetings
Management Team Executive Director
Executive Director Director ES Education
Executive Director
Management Team Executive Director
28
IMPLEMENTATION
The ES educational philosophy is realised in an effective, inclusive and learnercentred manner that is supported by a high-quality culture.
Standard Measurements Responsible Accountable
Online Learning Experience Principles guide course design.
2.1
THE ES PHILOSOPHY IS TRANSLATED INTO POLICY ACTIONS
2.2 ES QUALITY ASSURANCE SYSTEM IS IN PLACE
Range of policies in place:
• LEARN regulations
• Privacy statement
• T&Cs
• Licensing policy.
QA handbook.
ES Policy Advisor Management Team
Director ES Education
Institutional QA Cycle in place. ES QA Officer Director ES Education
2.3
PROCESSES ARE IN PLACE TO SUPPORT THE SYSTEMATIC DELIVERY OF ES EDUCATION AND SERVICES
Directory with processes.
Manager Education Support Manager Business Development
Executive Director
2.4
PROCESSES ARE IN PLACE TO COLLECT AND COLLATE DATA AND LEARNERS’ FEEDBACK TO IMPROVE EDUCATIONAL SERVICES
Course evaluation process. Portfolio review process. Research process.
Manager Education Support Manager Business Development
Executive Director
2.5
STAKEHOLDERS KNOW ABOUT AND ADHERE TO THE ES QUALITY CULTURE
ES quality values and definition created and shared.
Quality survey data and feedback.
ES quarterly meeting dedicated to quality culture.
Management Team Executive Director
29
EVALUATION AND MONITORING ES systematically evaluates whether the intended policy objectives relating to educational quality are achieved.
Standard Measurements
Responsible Accountable
3.1
THE ES STRATEGIC PLAN IS REVIEWED AND PERIODICALLY UPDATED FOR ITS CONTINUING RELEVANCE
ES Strategic Plan review process
Management Team
Executive Director
3.2
QUALITY METRICS ARE DEFINED AND REVIEWED ON A FREQUENT BASIS
Quality Standards & Indicators review process.
QA Officer
QA Board
Policy Advisor
Director ES Education
3.3
APPROPRIATE MEASUREMENT AND EVALUATION ACTIVITIES ARE INTEGRAL PART OF THE ES PROCESSES
Measurements used: learner surveys and other stakeholders, as needed. Dashboards evaluation activities:
• ES Annual Report to faculties and EB
• Annual evaluation (EB and ES-MT)
• ES Institutional evaluation
Management Team
Executive Director
3.4
THE CONTINUOUS IMPROVEMENT OF EDUCATION AND SERVICES IS ONE OF THE CONCRETE OUTCOMES OF THE ES EVALUATION PROCESSES
Continuous improvement section included in QA Annual Report.
Process Managers
Director ES Education Executive Director
3.5
EVALUATION RESULTS ARE PUBLISHED AND ACCESSIBLE TO RELEVANT STAKEHOLDERS.
ES reporting structure and documentation (reports).
Administrative Officer
Executive Director
30
DEVELOPMENT
ES focusses on the continuous improvement of its education, innovation and research.
Standard Measurements
Responsible Accountable
4.1
RESULTS OF THE ES QUALITY ASSURANCE ACTIVITIES ARE DISCUSSED ON A CYCLICAL BASIS AND AN IMPROVEMENT PLAN IS PUT IN PLACE
QA Board Annual Report.
QA Board Secretary Administrative Officer
Chair of QA Board Executive Director
4.2
BUDGET IS AVAILABLE FOR THE CONTINUING DEVELOPMENT OF ES STAFF AND FOR THE CONTINUOUS IMPROVEMENT OF TECHNOLOGIES AND SERVICES
Budget allocated for professional development, technologies and services.
Management Team
Executive Director
4.3
BEST PRACTICE ON CONTINUING EDUCATION AND LIFELONG LEARNING IS CAPTURED AND SHARED
Resources are published on ES website/Hub. Recurring item on all teams’ meeting agendas.
Communication Manager Education Support Team
Manager Education Support Manager Business Development
31
IMPACT
Continuing education impacts TU Delft and beyond.
Standard Measurements Responsible Accountable
5.1
ES CONTRIBUTES TO IMPROVEMENTS IN CAMPUS EDUCATION AND VICE VERSA
Report on impact on campus survey. Webinars and workshops dedicated to campus teaching staff.
Management Team Executive Director
5.2
ES CONTRIBUTES TO THE CONTINUOUS DEVELOPMENT OF THE LEARNER
Learners survey and feedback.
Management Team
Executive Director Director ES Education
5.3
ES ANNUAL INNOVATION AND RESEARCH BUDGET IS USED TO IMPROVE CONTINUING AND CAMPUS EDUCATION
ES budget allocation for innovation and research is in place
5.4
ES CONTRIBUTES TO THE UN SDGS
Mapping between Portfolios and UN SDGs.
Manager Education Support Manager Business Development
Executive Director
Education Management Team ES Education Director
5.5
ES ESTABLISHES A STRONG NETWORK WITHIN INDUSTRY AS A PARTNER AND SOUNDING BOARD FOR CONTINUING EDUCATION
Education Advisory Board includes industry professionals.
Manager Business Development Executive Director
5.6
ES BECOMES A SUSTAINABLE ORGANISATION AND IS EMBEDDED WITHIN THE UNIVERSITY AND IN THE CONTINUING EDUCATION/ LIFELONG LEARNING ECO SYSTEM AND BEYOND.
ES has a permanent status within TU Delft . Yearly budget dedicated to ES.
ES has a LLL strategy.
Management Team Executive Director
32
Portfolio Standards
PORTFOLIO STRATEGY
The ES portfolio strategy should encompass themes based on the expertise of the University and on learners’ demand.
1.1
THE ES PORTFOLIO THEMES ALIGN WITH THE UNIVERSITY’S STRATEGIC THEMES CONTRIBUTING TO FINDING SOLUTIONS TO SOCIETAL AND MARKET CHALLENGES
Checkbox over strategic alignment in Portfolio review.
Academic Portfolio Directors
Portfolio & Product Manager
1.2
ES HAS A PORTFOLIO STRATEGY FOR EACH THEME WITH A RESPONSIBLE PERSON (ACADEMIC PORTFOLIO DIRECTOR) IN PLACE
1.3
EACH PORTFOLIO THEME STRATEGY ADDRESSES COHERENCY AND ALIGNMENT IN ITS PORTFOLIO OFFERING
Portfolio strategy. Portfolio review process explores both angels:
• academic
• market relevance. Internal audit evaluation.
Portfolio review process takes into consideration alignment and coherency.
Academic Portfolio Directors
Portfolio & Product Manager
Academic Portfolio Directors
Portfolio & Product Manager
Director ES Education
Director ES Education
Director ES Education
1.4
ES HAS A CLEAR MANAGEMENT STRUCTURE THAT ENABLES IT TO SUPPORT ITS PORTFOLIO STRATEGIES
Internal audit evaluation addresses adequate management structure.
Manager Business Development Director ES Education
1.5
ES ALLOCATES SUFFICIENT RESOURCES TO EFFECTIVELY SUPPORT THE PORTFOLIO STRATEGY
Budget (multi-year plan and yearly).
Internal audit evaluation. Manager Business Development Director ES Education
33
Standard Measurements Responsible Accountable
EVALUATION AND MONITORING
The ES Portfolio is systematically evaluated for educational coherency and relevance.
Portfolio & Product Manager
2.1
THE ES PORTFOLIO STRATEGY IS REVIEWED AND PERIODICALLY UPDATED FOR ITS CONTINUING RELEVANCE
Portfolio strategy review process.
Manager Business Development
Academic Portfolio Directors
Director ES Education
2.2
LEARNERS’ ANALYTICS AND MARKET INFORMATION ARE USED TO IMPROVE THE PORTFOLIO STRATEGY
Market research is regularly conducted. Portfolio review.
Portfolio recommendations from edX.
Manager Business Development Director ES Education
2.3
APPROPRIATE MEASUREMENT, EVALUATION PROCESSES AND ACTIVITIES ARE EMBEDDED AT PORTFOLIO LEVEL.
Portfolio review. Portfolio strategy review.
DEVELOPMENT, ACCESS AND REACH
Portfolio & Product Manager
Manager Business Development
Director ES Education
The ES Portfolio is broad and serving a diverse target audience across the globe.
3.1
THE ES PORTFOLIO PROVIDES UP-TO-DATE AND STATE-OF-THE-ART EDUCATION
Portfolio review process.
Portfolio & Product Manager
Academic Portfolio Directors
3.2
THE ES PORTFOLIO ENGAGES BOTH INTERNAL TU DELFT STUDENTS AND EXTERNAL LEARNERS ENCOURAGING CONTINUING EDUCATION/ LIFELONG LEARNING
Annual Report. Reuse of OER in accredited and continuing education. Input from lecturers gathered in evaluation
3.3
THE ES PORTFOLIO ATTRACTS AND CATERS FOR DIVERSE, YET THE RIGHT KIND OF LEARNERS.
Diversity is taken into account in the course design and promotion.
Net Promoter Score and course grade; learner satisfaction.
Portfolio & Product Manager Director ES Education
Learning Developers
Marketers
Manager Education Support
34
Standard Measurements Responsible Accountable
Standard Measurements Responsible Accountable
IMPACT
The ES portfolio impacts TU Delft and beyond.
4.1
EDUCATION MATERIALS ARE USED FOR CAMPUS EDUCATION (AND VICE VERSA) AND BEYOND TU DELFT
Annual Report. Re-use of OER in accredited and continuing education. Input from lecturers gathered in evaluation.
4.2
THE ES PORTFOLIO CONNECTS TO THE HUMAN CAPITAL AGENDA OF THE UN SDGs
Portfolio Strategy is informed by market and trend analysis. Themes relate to the UN SDGs
Portfolio & Product Manager
Faculty Coordinator
Director ES Education
Portfolio & Product Manager
Academic Portfolio Directors
4.3
ES IS VISIBLE AND ACTIVE IN THE ECOSYSTEM OF CONTINUING EDUCATION/ LIFELONG LEARNING AT NATIONAL AND GLOBAL LEVELS.
ES participates actively in (inter) national LLL activities, especially with regards to the portfolio themes.
ES is promoted and visible in (inter) national platforms.
Manager Education Support
Manager Business Development
Academic Portfolio Directors
Academic Portfolio Directors
35
Measurements Responsible
Standard
Accountable
Course Standards
COURSE DESIGN AND DEVELOPMENT
Courses are designed and developed to provide a high-quality, stimulating, and engaging learning experience.
1.1
EACH COURSE HAS A QUALIFIED COURSE TEAM WITH THE EXPERTISE REQUIRED TO ACHIEVE THE COURSE GOALS
1.2
THE COURSE DEVELOPMENT PROCESS ENSURES COURSE TEAMS ARE PROVIDED WITH RELEVANT RESOURCES
Addressed at kick-off meeting and division of tasks.
In re-runs: Course Team completeness/needs are checked as part of the evaluation.
Roadmap for creating an online course is in place and followed.
We have a process in place to review and update trainings and resources.
Responsible Lecturer
Product Portfolio Manager
Learning Developers Coordinator
Manager
Education Support
1.3
THE ES ONLINE LEARNING EXPERIENCE PRINCIPLES GUIDE THE COURSE DESIGN
Blueprint template reflects OLE principles. Alignment of intentions and results (with respect to OLE) by Course Team at evaluation stage. Reflection included in Evaluation Summary.
1.4
THE LEARNING OBJECTIVES ARE ALIGNED WITH SPECIFIC ASSESSMENTS AND LEARNING ACTIVITIES FOR THE GIVEN TARGET AUDIENCE
Story board follows constructive alignment.
Learning Developer Manager Education Support
Course Team
Responsible Lecturer
1.5
COURSES ARE DEVELOPED BASED ON OPEN PRACTICES, WHERE POSSIBLE
MOOCs are designed and created for reuse, meaning that course material created has an open license. Content re-used (and modified) is available in open access and correctly attributed.
1.6
THE COURSE DESIGN FOLLOWS ACCESSIBILITY AND USABILITY GUIDELINES
Accessibility checklist is used for course development.
Course Team
Learning Developers Learning Technology Team
Process Manager Open Education
Manager Education Support
36
Standard Measurements
Responsible Accountable
COURSE CURRICULUM
Courses enable learners to integrate knowledge and professional skills in the fields of Science, Engineering, and Design.
Standard Measurements
Responsible Accountable
2.1
COURSES FIT WITHIN THE FIELDS OF SCIENCE, ENGINEERING AND DESIGN
Courses that fit in one of the Portfolios themes have priority - tender process check.
Responsible Lecturers
Faculty Coordinators
Manager Business Development
2.2
THE COURSES’ ACTIVITIES, MATERIALS, AND LEARNING EXPERIENCE ENABLE LEARNERS TO ACHIEVE THE INTENDED LEARNING OUTCOMES
Learning outcomes are part of the Story Board together with activities and course material. Pass rate analysis.
Post-survey.
2.3
THE COURSES’ SYLLABUS OUTLINES THE COURSE OVERVIEW, LEARNING OBJECTIVES, STUDY LOAD, AND ASSESSMENT DETAILS SO THAT THE LEARNER KNOWS WHAT TO EXPECT
Information is stated in the Syllabus and course outline.
Course Team
Responsible Lecturers
Course Team Learning Developers
Responsible Lecturers
2.4
COURSES MATCH THE DIVERSE TARGET AUDIENCES IN ACCORDANCE WITH THE ES DIVERSITY AND INCLUSION STATEMENT
Course content is designed for broad accessibility.
Target audience in the About Page.
Post-survey.
Course Team Learning Developers
Manager Education Support
37
ASSESSMENT
Courses have an adequate assessment system in place according to the type of course.
Standard Measurements
Responsible Accountable
3.1
LEARNING OUTCOMES ARE ASSESSED USING A BALANCE OF FORMATIVE AND SUMMATIVE ASSESSMENTS APPROPRIATE TO THE CURRICULUM
Assessment information and type is clearly stated in the Syllabus.
Learning taxonomies are used to determine the right level and types of assessment for the course.
3.2
ASSESSMENTS ARE AUTHENTIC, VARIED AND INCLUSIVE
Review assessments created during Review and Reality Check. Post-survey.
3.3
ASSESSMENT CRITERIA AND RESULTS ARE CLEARLY AND TIMELY COMMUNICATED TO LEARNERS
Rubrics and grading tables in the course are available and comprehensive.
Course Team Responsible Lecturers
Course Team Responsible Lecturers
Course Team Responsible Lecturer
3.4
LEARNERS MAY BE ALLOWED TO DO RETAKES WITH THE APPROVAL OF COURSE TEAMS WHERE APPLICABLE
LEARN and T&Cs in place, which allow retakes.
QA Officer Policy Advisor Director ES Education
3.5
THERE IS A PROCESS IN PLACE TO HANDLE DISPUTES AND FRAUD CASES
LEARN and T&Cs in place. Fraud and complaints process described in QA Handbook. Fraud and complaints repository.
3.6
THE QUALITY ASSURANCE BOARD UPHOLDS THE QUALITY STANDARDS WITH REGARDS TO ASSESSMENT AND CERTIFICATION
Certificates are signed on behalf of the QA Board. Disputes about assessment results and fraud are addressed to the QA Board.
QA Officer Policy Advisor Director ES Education
QA Officer Chair QA Board
38
COURSE DELIVERY
Course delivery methods encourage learners’ engagement and active learning through consistent moderation and feedback.
Standard Measurements
Responsible
Accountable
4.1
COURSE TEAM MEMBERS IN CHARGE OF MODERATION ARE FULLY TRAINED BEFORE STARTING TO MODERATE COURSES
TAs on-boarding guide in place.
Building And Moderating (BAM) an online course training is mandatory for staff before the course starts.
4.2
COURSE MODERATION IS CONSISTENT AND OF GOOD QUALITY
E-moderation guidelines, training and resources are available.
Post-survey.
4.3
EACH COURSE INCLUDES (IN) FORMAL FEEDBACK TO HELP LEARNERS IMPROVE THEIR PERFORMANCE
Moderation guidelines include instructions on how to provide feedback to learners.
Post-survey question to learner about feedback.
LET Coordinator
Faculty Coordinator
Learning Developers
Course Team Course Moderator
Responsible Lecturer
4.4
LEARNERS ARE INFORMED OF AND HELD ACCOUNTABLE FOR ADHERING TO THE ES CODE OF CONDUCT, AND TO THE FORUM AND COLLABORATION GUIDELINES
All learners are required to review T&Cs and Code of Conduct upon enrolment.
Discussion Forum Guidelines, Collaboration Guidelines and Honour Pledge in place and embedded in course shell and syllabus.
Course Team Course Moderator
Responsible Lecturer
LET Coordinator
Responsible Lecturer LET Coordinator
Manager
Policy Advisor
Learning Developers
Business Development Manager Education Support
4.5
COURSE DELIVERY IS EVALUATED THROUGH THE ANALYSIS OF PLATFORM DATA, INPUT FROM LEARNERS ON THEIR EXPERIENCE, AND FEEDBACK FROM THE COURSE TEAM
Learners post-survey.
Platform data analysis of learners engagement
Moderation feedback log/ report or in course hidden section.
Data Analyst
Course Team
Learning Developers
Manager Education Support
39
MONITORING AND EVALUATION
Courses are systematically evaluated for their educational quality and relevance. Standard
Accountable
5.1
THERE IS A QA CYCLE IN PLACE FOR CONTINUOUS IMPROVEMENT
Course evaluation process. QA Officer Manager Education Support
5.2
LEARNERS’ DATA AND COURSE TEAM FEEDBACK CONTRIBUTE TO THE IMPROVEMENT PLAN
Evaluation report and survey results. Course Team feedback Learning developers’ observations. Moderation feedback report.
Data Analyst
Course Team Learning Developers
Manager Education Support
5.3
COURSES ARE REGULARLY REVIEWED AND IMPROVED (INCLUDING IN OPEN COURSEWARE) TO REMAIN RELEVANT FOR LEARNERS
Evaluation process and improvement plan after 1st run and every 3 runs after the 2nd
Portfolio & Product Manager
Manager Education Support
5.4
THE QUALITY ASSURANCE BOARD UPHOLDS THE QUALITY OF EDUCATION BY MONITORING THE COURSE EVALUATION PROCESS
QA Board regularly requests to be provided documentation concerning course evaluation process
QA Officer/ QA Board Secretary
QA Board Chair
40
Measurements Responsible
TECHNOLOGY Technology adequately supports the courses’ learning outcomes.
Standard Measurements
Responsible Accountable
6.1
A VARIETY OF TECHNOLOGIES SUPPORTS THE ACHIEVEMENTS OF THE LEARNING OUTCOMES
Assessed during Review and Reality Check and/or Story Board, includes type and variety of technology
Learning Developers Course Team
Manager Education Support
6.2
COURSE TEAMS ARE SUPPORTED IN THE DEVELOPMENT AND USE OF TECHNOLOGIES
ES has a process in place for the development of new technologies. Trainings and resources available on the use of technologies. edX Support Team is in place.
Learning Technology Team LET Coordinator
Manager Education Support
6.3
LEARNERS ARE ADVISED OF THE MINIMUM TECHNICAL REQUIREMENTS NEEDED TO BE ABLE TO FOLLOW A COURSE
System requirements information and general restrictions for all courses are indicated in ES website. Course-specific additional technical requirements are indicated in the About Page.
Learning Technology Team Course Team
Manager Education Support
6.4
LEARNERS ARE INFORMED ON HOW THEIR DATA AND PRIVACY ARE PROTECTED
ES and 3rd party platform Privacy Statement, and electronic consent in place. Policy Advisor Executive Director
41
Annex 2: Example of an evaluation instrument: Post-survey
Q196 The course has ended. We would like to thank you for your participation and contribution to its success! We are very interested to know about your experience during this course. With your feedback we can improve our courses, and enable more people to access, enjoy, and complete them.
This survey should only take 5-10 minutes to complete. Participation is completely voluntary and all information will be treated confidentially.
The data will be reported only in aggregate form and no individual will be identified. Your answers here, combined with your edX course data, will be used for evaluation purposes, and potentially in scientific research related to online education, which is why we need your permission.
Please feel free to contact us if you have specific questions about this survey, by sending an email to eduanalytics@tudelft.nl.
Thank you very much for helping us improve our courses!
Q198 This survey is in accordance with the edX Privacy Policy.
Q1edX ELECTRONIC CONSENT: Please select your choice below. Selecting the “agree” button below indicates that:
You have read and understand the above information. You voluntarily agree to participate. You give us permission to use your input for scientific purposes. You are at least 16 years of age.
Agree (1)
Disagree (0)
Q1 Based on your entire experience with TU Delft Online Learning, on a scale from 0-10, how likely are you to recommend us to a friend or colleague?
Q2 Please help us by explaining your rating.
Q2.1 Since the start of the course, how would you describe your participation level? Choose the answer that best applies to your situation.
In order for the survey to unfold correctly you need to answer this question.
• I did not participate in this course at all (1)
• I did not participate in this course, but I did browse around a little bit (2)
• I only looked at specific parts of the course that I was interested in (3)
• I participated in the course, but I stopped participating along the way (4)
• I participated in the course until the end (5)
42
• 0 (0) • 1 (1) • 2 (2) • 3 (3) • 4 (4) • 5 (5) • 6 (6) • 7 (7) • 8 (8) • 9 (9) • 10 (10)
Q2.2.1 Could you please describe the reason(s) why you did not start the course? ___________________________________________________________________________
Q2.2.2 Could you please describe which specific parts of the course you were interested in and why?
Q3 The next questions are about your overall experience and impression of the course. Q3.1 On a scale from 1 to 10, what overall grade would you give the course?
very poor, 10: excellent)
Q3.2 What did you find most valuable in the course? Please explain.
Q3.3 Which aspects of this course would you like us to improve? Please explain.
Q4 What is your current type of enrollment in the course?
• Audit - I am enrolled for free (1)
• Verified - I paid (2)
Q4.1 What additional value did you get from the verified track? ___________________________________________________________________________
Q4.2 How would you rate the following aspects of the course? The course was ...
Strongly disagree (1)
Somewhat disagree (2)
Neither agree nor disagree (3)
Strongly agree (5) Unique (Q4.5_1) Useful (Q4.5_2) Interesting (Q4.5_3)
Somewhat agree (4)
43
Very poor Excellent Overall grade: () 1 2 3 4 5 6 6 7 8 9 10
(1:
Q4.3 How would you rate the difficulty level of the course?
• Far too difficult (1)
• Too difficult (2)
• About right (3)
• Too easy (4)
• Far too easy (5)
Q4.4 How would you describe the amount of work required for the course?
• Far too little (1)
• Too little (2)
• About right (3)
• Too much (4)
• Far too much (5)
Q4.5 How would you describe the breadth of topics covered in the course?
• Far too narrowly (1)
• Too narrowly (2)
• About right (3)
• Too broad (4)
• Far too broad (5)
Q4.6 How would you describe the length of the course (i.e. number of weeks)?
• Far too short (1)
• Too short (2)
• About right (3)
• Too long (4)
• Far too long (5)
Q5 The next couple of questions are about how you participated in the course and how you perceived different elements of the course.
Q5.1 On average, how many hours per week did you work on this course? This includes assignments, reading material, and video lectures. Hours 0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20
Q5.2 Which elements of the course did you use or participate in? Choose all that apply.
• Videos (1)
• Reading materials (2)
• Forums (3)
• Exercises, quizzes, assignments (4)
• Group work (5)
44
Q5.2.1 How satisfied were you with the following elements of this course?
Videos (Q5.2.1_x1)
Reading materials (Q5.2.1_x2)
Forums (Q5.2.1_x3)
Exercises, quizzes, assignments (Q5.2.1_ x4)
Group work (Q4.2.1_ x5)
Feedback by instructors (Q4.2.1_x6)
Possibility to work on your own project (Q4.2.1_x7)
Q5.2.2 How valuable were the following elements of this course? Not at all valuable (1) Somewhat valuable (2) Moderately valuable (3) Valuable (4) Very valuable (5)
Videos (Q5.2.1_x1)
Reading materials (Q5.2.1_x2)
Forums (Q5.2.1_x3)
Exercises, quizzes, assignments (Q5.2.1_ x4)
Group work (Q4.2.1_ x5)
Feedback by instructors (Q4.2.1_x6)
Possibility to work on your own project (Q4.2.1_x7)
45 Very dissatisfied (1) Somewhat dissatisfied (2) Neither satisfied nor dissatisfied (3) Somewhat satisfied (4) Very satisfied (5)
Q5.2.3 Why did you not use or participate in ${Q4.2/ChoiceGroup/UnselectedChoices}?
Q5.3 What was the biggest challenge in completing this course?
• Using the learning platform (1)
• Meeting the deadlines for assignments and exams (2)
• Grasping the course content (3)
• Allocating sufficient time and combining it with other life obligations (4)
• Other: (5)
• I didn’t experience any challenges (6)
Q5.3.1 At the beginning of the survey you said that you participated in the course, but stopped participating along the way. Why did you not participate in the course until the end? Choose the answer that applies the most to your situation.
• Due to personal obligations. (1)
• Due to an unexpected change in available time. (2)
• Due to a general lack of time. (3)
• The course was not what I expected. (4)
• The course was too difficult for me. (5)
• I found a better course somewhere else. (7)
• Other: (6)
46
_____________________________________________________________
_____________________________________________________________
Annex 3: List of accredited courses
Online Academic Courses – Course accreditation as part of the Master Aerospace Engineering at TU Delft
• Aeroelasticity
• Modeling, Simulation and Application of Power and Propulsion Systems
• Helicopter Performance, Stability and Control
• Non-linear Modeling
• Design of Lightweight Structures I: Composites & Metals
• Fatigue of Structures & Materials
• Linear Modeling (including FEM)
• Advanced Dynamics
• Rotor and Wake Aerodynamics
Accreditation period: 21 January 2020 to 01 November 2025.
Solar Energy Engineering MicroMasters® – Course accreditation as part of the TU Delft Master Sustainable Energy Technology and the Master Electrical Engineering.
The accreditation periods are respectively: 30 November 2017 to 01 November 2023, and 31 August 2017 to 01 May 2023.
Aerobic Granular Sludge Technology for Wastewater Treatment – Accredited by Chartered Institution of Water and Environmental Management (CIWEM).
Accreditation period: 21 November 2021 to 20 November 2023.
Nanofiltration and Reverse Osmosis in Water Treatment – Accredited by Chartered Institution of Water and Environmental Management (CIWEM).
Accreditation period: 21 November 2021 to 20 November 2023.
47
TU Delft Extension School for Continuing Education
Since 2013, our online courses and programs help thousands worldwide to make an impact on areas of great relevance to the environment and society. By delivering expert knowledge and applicable skills for engineers, we support working professionals to stay on top of their game – whether they are looking to ch ange career, broaden their skillset, or acquire further academic qualifications.
www.tudelft.nl/extension-school extension-school@tudelft.nl
@TUDelftOnline
Building 32, Landbergstraat 15, 2628CE Delft, The Netherlands
Please attribute TU Delft Extension School CC-BY-NC-SA.
