Introduction
EER Evaluators build close relationships with providers and through their roles, are witness to lots of good practice in action.
This insights paper shares stories from our evaluators of provider good practice across four tertiary education indicator (TEI) categories. Many of these stories highlight the importance of collaboration between NZQA and providers, and share how providers work to continually improve their practices.
During the EER process, tertiary education providers use evidence and accompanying analysis to answer six key evaluation questions, which are aligned to NZQA’s five tertiary evaluation indicator (TEI) categories:
- Programmes match needs
- Achievement and outcomes
- Student engagement
- Governance and management
- Compliance
In a companion to this paper, available from Insights papers, we take a deep dive into the ‘programmes match needs’ TEI category, and good practice themes from a sample of 94 Category 1 EER reports published between April 2019 and March 2022. In this paper we share stories of provider good practice from across the remaining TEI categories.
Through sharing these stories, NZQA hopes to offer up a repertoire of positive practice for anyone involved in tertiary education. These are not meant as one-size-fits-all solutions. They do not offer a blueprint for success in EER. Rather, these stories emphasise the flexibility and diversity of real-world providers. They offer practical possibilities that other organisations may adapt or modify as appropriate, on behalf of their students.
Download this paper as a PDF
On this page
Achievement and outcomes
The TEI category ‘achievement and outcomes’ consists of six indicators, aligned to the following two key evaluative questions: ‘How well do students achieve?’ and ‘What is the value of the outcomes for key stakeholders, including students’.
The indicators that guide providers to succeed in this category are:
- Students acquire useful skills and knowledge and develop their cognitive abilities.
- Students complete courses and/or gain qualifications.
- Students gain relevant employment and/or engage successfully with further study.
- Students apply new skills and knowledge and contribute positively to their local and wider communities.
- Students improve their well-being and enhance their abilities and attributes.
- Communities’ and iwi bodies of knowledge are created, developed, and advanced.
‘Achievement and outcomes’ in action: Foundational skills training provider (Provider A)
Provider A is a private training establishment focused on offering foundation level learning. Their core business is job placement programmes contracted by the Ministry of Social Development, and their primary focus is supporting learners to gain the key knowledge, skills, confidence, and attitude they need to secure and retain sustainable employment. A high proportion of Provider A’s ākonga identify as Māori and Pasifika.
Provider A’s staff produce ‘student success stories’ for each cohort of ākonga they work with. These stories highlight the achievement and outcomes of their learners, and include some information about each student’s background, how they came to study at Provider A, a description of the challenges they overcame, and a dive into what each ākonga has accomplished following their studies.
These stories clearly demonstrate the value of Provider A’s service, and the usefulness of the skills and knowledge developed by ākonga during their time with the provider. The stories demonstrate that learners were happy with the knowledge they gained, and the trainers they worked with.
One of Provider A’s key programmes is the New Zealand Certificate in Foundation Skills in Security (Level 2), a 19-week programme that provides youth, including some at-risk youth, with an opportunity to improve communication skills and self-confidence. This training towards security licences, with wrap-around individual support from Provider A’s staff, helps most graduates to gain meaningful employment. Their work-focused programme enables graduates to apply their learning and work more safely and effectively immediately upon employment.
Provider A learners are motivated, positive about their learning, and well-supported by trainers, some even long after finishing their enrolments. Over 60% of ākonga complete the programme, and more than 90% either gain employment or continue to further studies. Interviews and documents provided during NZQA monitoring in June 2021 showed that 12 out of 15 learners gained employment upon completion of their studies.
Provider A is strongly employment focussed and committed to ensuring that students and employers receive high value from the training and services they deliver. Their ongoing relationships with employers are valuable for learners, and the fact that employers repeatedly employ graduates from Provider A over several years is evidence of the perceived value of the training they provide.
‘Achievement and outcomes’ in action: Pathways training provider (Provider B)
Provider B is a family-owned training provider, focused on helping young Māori, Pasifika, and adult second-chance learners in their career journeys. They offer foundational, pathway training programmes in supported learning, retail, hospitality, youth training and English language. Provider B is one of the largest tertiary providers to students with special educational needs.
Provider B’s student achievement is good, and their programmes have clear relevance for their learners, helping them to gain the skills and knowledge required in work placements. They include their tutors in the interview and enrolment process to ensure learner placements are well aligned, and provide extra support to students who have identified learning difficulties.
Provider B knows what adds value and keeps track of all graduates and their outcomes. There is follow-up contact through graduates’ visits to campus and phone surveys.
Tutors accompany students to work placements to help provide formative feedback. This is particularly important for learners who are young and lack life experience or confidence. Tutors make sure learners are fully prepared before they start their work placement.
Provider B’s learners highly value the outcomes they gain through their training. English language graduates develop self-confidence and feel enabled to engage more in the community, whilst supported learning students value their personal growth in skills and abilities which help with being more autonomous. Hospitality students frequently progress to higher levels of study, and many gain work immediately after completion of their studies with Provider B.
Student engagement
The TEI category ‘student engagement’ consists of eight TEIs, aligned to the key evaluative question: ‘How effectively are students supported and involved in their learning?’.
The indicators that guide providers to succeed in this category are:
- Student learning goals are well understood.
- Comprehensive and timely study information and advice is provided to assist students pursue their chosen pathways.
- Responses to the well-being needs of students are appropriate.
- The learning environment is inclusive.
- Policies and procedures minimise barriers to learning.
- Students have opportunities to apply knowledge and skills in a variety of contexts.
- Students are supported to establish effective social and academic support networks.
- Students are provided with useful and timely feedback on their progress.
‘Student engagement’ in action: Creativity training provider (Provider C)
Provider C provides training in creativity processes, expressed through art mediums, to adult and second chance learners. They pride themselves in their learner-centred approach, where learners collaborate with mentors and tutors throughout their studies.
All prospective students are interviewed to ensure they are placed in the most suitable programme for them. They are provided with comprehensive study information via the student handbook, which outlines all programme information, study expectations and support processes, and a beginner’s kit with basic resources which sets learners up with the tools they need to succeed in their studies.
At Provider C, each learner is recognised as a unique adult learner with prior experience and skills, supported through self-directed learning. Their learners choose which courses would be most beneficial towards achieving the graduate profile outcomes. This means that the learners themselves form a part of pre-assessment moderation of each assessment activity, in that they are choosing how to meet assessment criteria within their projects.
This approach, through treating each learner as unique with individual learning needs allows Provider C to ensure the learning environment is inclusive, and students have opportunities to apply knowledge and skills in a variety of contexts. Provider C empowers their learners to assess which skills they already hold, and which skills they need to work on to achieve the outcomes designed within their creativity projects. Because of this approach, their learners are well-engaged and motivated to attend courses.
Provider C has regular, meaningful engagement with their students to allow opportunities for feedback and discussion of any challenges their students are facing. They work quickly to respond to any student suggestions to improve the learning and support they offer, and their tutors and mentors are well versed in developing the creative process to help students move past blockages and develop their skills.
‘Student engagement’ in action: Creativity training provider (Provider D)
Provider D is a small private training establishment which specialises in working with students who have been unsuccessful in traditional school systems. Many of their learners are under 20 years old and have had disrupted schooling, sporadic attendance, and often leave school early. Due to the unique demographics of their learners, student engagement is particularly crucial for the success of the organisation.
Provider D is dedicated to removing barriers to learning and providing appropriate responses to the well-being needs of their students, ensuring that they are well-supported and motivated to achieve success in their learning. An unexpected event during an EER evaluation provided an opportunity for the organisation to demonstrate their commitment to their learners and showcase the importance of student engagement.
During the evaluation, a student who was expected to be interviewed arrived late and accompanied by a police officer after an incident. Instead of halting the evaluation, Provider D used this opportunity to demonstrate their approach in action. The organisation’s manager remained close to the student, encouraging them to open up about what had happened in a supportive environment, and took the initiative to help the student make amends for their actions. The manager also worked closely with the student’s parents to explain the situation.
This incident highlighted the importance of student engagement and demonstrated the support that Provider D provides to their learners outside of formal learning. The time and commitment given to this learner was common practice for Provider D as they know the positive impact that this kind of support has on learners’ education experience. This level of engagement is crucial in providing second chance learners with the best opportunities to succeed in their studies.
The support that Provider D provided to the student was invaluable. In an interview with the learner, they confirmed that without such support, they would likely have been discouraged in their learning and would have missed out on opportunities to gain qualifications and acquire valuable skills required for the workforce. This story emphasises the importance of wrap-around pastoral support in student progress and reaffirms the value of providing additional time and support to keep learners engaged and on track.
‘Student engagement’ in action: Fishing industry training provider (Provider E)
Provider E is a small private training establishment focused on training new entrants into the New Zealand fishing industry. Many of their learners come from disadvantaged backgrounds and often struggle with discipline and commitment to education.
Through effective engagement with their students, Provider E recognised that their students would benefit from having a stable and structured learning environment to better support their learning. To better support their students’ learning, Provider E purchased an accommodation facility where all learners stay while enrolled in their programs. This facility enables the learners to stay focused and receive dedicated and individual support whilst studying.
The staff at Provider E are committed to creating an inclusive learning environment that feels like a family, with a team focus. This type of setup is appropriate for their students, and helps them meet the graduate profile outcomes of their qualifications.
Working in the fishing industry often involves extended periods away at sea. Discipline, commitment, and endurance are essential qualities that the industry seeks in its staff. Provider E understand that their students learning goals are primarily focused on finding meaningful employment in the industry. As a result, they have designed their programmes to help prepare the learners for real-life experiences with exposure to boats and the discipline of being away from family and friends.
Provider E takes a holistic approach to learning and prioritises their students’ well-being. In addition to providing training programs, the school also offers a range of educational activities that support learners through their journey. Students gain life skills, such as how to stay healthy and exercise, which are crucial for maintaining good mental and physical health onboard a vessel. These skills provide a solid foundation for graduates to be successful and ensure that the industry gains competent, reliable, and dedicated staff.
Governance and management
The TEI category ‘governance and management’ consists of nine TEIs, aligned to the key evaluative question: ‘How effective are governance and management at supporting educational achievement?’
The indicators that guide providers to succeed in this category are:
- Organisational purpose and direction is clear.
- Organisational academic leadership is effective.
- Sufficient resources are allocated to support learning, teaching and research.
- Data analysis is used effectively throughout the organisation.
- Recruitment and development of staff is effective.
- Staff are valued.
- The education organisation anticipates and responds effectively to change.
- Innovation, responsiveness and continuity are balanced.
- The TEO operates a sustainable business model, which is aligned to its educational purpose.
‘Governance and management’ in action: Employment services training provider (Provider F)
Provider F is a small private training establishment delivering youth training, workplace literacy and numeracy, and employment services programmes for MSD.
Provider F’s organisational academic leadership is effective, with the management team having built an engaged and collaborative team that is fully committed to using data and stories to make meaningful improvements. Their team have implemented a collaborative approach to management that uses real-time data and reporting effectively to track, monitor, and understand learner achievement.
Data analysis is used effectively by staff across Provider F, with the whole organisation involved in generating, analysing, using, and understanding achievement and performance data to make meaningful changes to daily practice. They know what is happening, why and what it means for improvements to the programme and student achievement.
They hold informal daily and weekly meetings, have in-depth reporting practices, and make use of visual data boards to track progress against important goals such as attendance and achievement. This results in a clear picture of organisational performance for staff, so they are each aware of what they need to do to reach the desired outcomes. Provider F’s strong focus on data-driven decision-making demonstrates the organisation’s commitment to anticipating and responding effectively to change.
There is limited expertise in the organisation for complex data analysis or high-tech reporting, and yet Provider F is using data and information effectively to drive business performance. Staff have bought into the organisational values and business processes to work together to ensure strong achievement for learners and Employment Services participants.
‘Governance and management’ in action: Early childcare training provider (Provider G)
Provider G is a private training establishment offering early childcare education courses, which underwent significant structural changes. Through effective and responsive management, they improved internal practices and learner outcomes.
Despite being a nationwide organisation, some geographic areas were disproportionately served in terms of resources and personnel. There were workload concerns with one manager responsible for most of the organisation’s administrative work and moderation oversight of 120 assessors, as well as some issues regarding the consistency of their moderation practices across the country.
Management at Provider G self-identified the need for changes in the organisation’s resourcing, marking guidance, internal moderation system, and the management structure. In response to these identified areas for improvement, they took action to improve their organisational performance. They reduced the number of assessors to seven - all of whom were trained in assessment and moderation - and appointed two senior assessors to provide feedback to others as part of the moderation process.
Provider G worked with NZQA to gain a better understanding of how their moderation practices could be improved. Management used NZQA’s internal moderation report monitoring outcomes to evidence the need for investment in the professional development of their staff, streamlining the organisation’s reporting structure, and updating and altering the moderation system. External contractors were hired to facilitate the process, and samples of marking guidance and rubrics were created and reviewed by NZQA, resulting in improved staff cohesion and consistent and fair assessment of all learners throughout the country. Organisation policies were also revised to reflect the new system and ensure clear lines of action and accountability if moderation identified areas of concern.
The successful improvements at Provider G were made possible through effective governance and management that recognised the need for change, and who were willing to take appropriate action to enhance their practices.
Compliance
The TEI category ‘compliance’ consists of three TEIs, aligned to the key evaluative question: ‘How effective are governance and management at supporting educational achievement?’.
The indicators that guide providers to succeed in this category are:
- Policies and practices are legal and ethical.
- The TEO has effective compliance management processes.
- Relevant legislation, rules and regulations are complied with.
‘Compliance’ in action: Key competencies training provider (Provider H)
Provider H delivers foundation-level training to support second-chance youth and adult students in their journey towards gaining employment or entering further study. The organisation focuses on literacy and numeracy, key competencies development and employability skills.
Recently, Provider H participated in an NZQA programme monitoring activity, which aimed to ensure compliance with the Programme Approval and Accreditation Rules. The monitoring activity focused on the NZ Certificate in Foundation Skills (L1) and aimed to evaluate the quality and relevance of learner credentials.
Provider H staff worked with NZQA to gain a better understand of the requirements, and the types of evidence they would need to provide. When they provided the required information, NZQA noted that many of Provider H’s documents, policies and processes had been recently updated. The provider explained that they had been concerned about their existing information-collation processes, and had set about reviewing their processes, programmes and outcomes. The monitoring activity enabled them to realise that the programme review process could be more rigorously managed to meet learner needs.
Through their self-review, Provider H identified gaps and weaknesses in their compliance management processes and set out action plans with clear timelines to address them. They outlined in detail how they were going to manage the issues on an ongoing basis, and the plans were found to be effective and appropriate in their context, which gave NZQA confidence in the programme review process.
This story demonstrates how a collaborative approach to quality assurance can result in positive outcomes for learners. The monitoring activity had prompted Provider H’s team to work with NZQA to gain a better understanding of the ongoing requirements for programme delivery, and ultimately resulted in the implementation of better compliance management processes within the organisation.
‘Compliance’ in action: Marine industry training provider (Provider I)
Provider I is a small private training establishment which works with students in the marine industry. The organisation’s management are well-embedded in the industry with personal experience and familiarity with the tools of the trade.
Through a consistency review1 of one of their qualifications, it was noted that despite informal stories of success, there was a lack of formal evidence to demonstrate that graduates of the programme had achieved and were applying the graduate profile outcomes of their qualification. This resulted in an initial ‘not sufficient’ rating, with the opportunity to submit further evidence.
Following this, Provider I reached out to NZQA for guidance on how to best demonstrate that their graduates were doing well. They were concerned that HR professionals in marine companies would not be able to provide the evidence they needed, as they wouldn’t be familiar with the work of Provider I graduates on boats. The conversation with NZQA helped the provider to realise that they were in charge of their own quality assurance processes, and that NZQA recognised that their staff would be best placed to decide for themselves how and from whom feedback was collected, and how it would inform their programmes, as they have existing connections and inside knowledge of the marine industry.
As a result of this conversation, Provider I became more engaged in the process and began putting quality assurance systems into place. When their submission of further evidence arrived, it showed the work they had done to find out what their graduates were doing and revealed support from industry. It was comprehensive and effective, and included changes they would make to the equipment used in the training, as a response to feedback. The submission came with an email describing how the provider had come to realise the importance of collecting feedback to inform improvements to their programme and had taken steps to make the system more efficient going forward.
As a result of this consistency review, the provider was empowered to take greater control of their own quality assurance and began to see the value of formally collecting feedback from graduates and stakeholders. This, in turn, will be likely to benefit both learners and industry.
Provider I’s story illustrates how a provider can be encouraged to see their own capability and potential to promote positive change within their organisation by implementing effective compliance management processes. The intervention showed the provider that NZQA acknowledged their expertise and skills, which can promote good engagement with a quality assurance process that will ultimately benefit their learners.
¹Consistency reviews are part of the assuring national consistency of graduate outcomes process.
Acknowledgements
NZQA would like to thank tertiary providers for participating in case studies and in the External Evaluation and Review process, insights from which form the basis for this paper.