Monitoring of tertiary education organisations' programme review processes: 2022 summary

Tukanga arotake | The monitoring process

This programme monitoring activity was focused on TEOs’ programme review processes and sought to answer the following key evaluative question (KEQ):

NZQA monitored the programme review processes of 19 TEOs delivering programmes leading to the New Zealand Certificates in Foundation Skills (Levels 1-2) and Study and Career Preparation (Level 3).

TEOs provided qualitative data for NZQA to complete a desk evaluation of the documents and NZQA also completed interviews with at least two staff members from each TEO, which included the programme leader and one tutor for the programme. The evaluation sought to provide assurance that TEOs are continuing to meet requirements for maintaining approval and accreditation, including:

• the effectiveness of the TEO’s ongoing programme review and monitoring processes and policies
• the currency and content of the programme
• the quality of outcomes for learners and other stakeholders

Please see Appendix 1 for a full list of the outcome criteria and evidence expectations.

Ngā kitenga | Monitoring findings

Ongoing programme review and monitoring

Areas of good practice:

• Most TEOs had clear, effective and documented processes, policies and guidelines for the review and monitoring of programme performance. These are formally reviewed and updated at least once annually and informally reviewed as and when the need arises. Some of these instances included, apportioning time at staff hui and especially during the 2020-2021 COVID-19 lockdown where TEOs pivoted towards online learning.
• TEOs who have scheduled annual and ongoing informal programme reviews, which are reflective in nature, had better outcomes for all stakeholders. These included a consistent application of the processes and policies which are re-enforced through effective action plans. Most of the TEOs rated as excellent/good scheduled programme reviews at the end of every teaching cycle which led to early identification of issues, challenges and gaps.
• TEOs who used effective documented action plans had a more effective programme review process. Effective documented action plans are those which state what improvements are needed, how the improvements will be achieved and measured, who is responsible, a timeline for completion and the desired outcomes.
• Some TEOs showed responsiveness to learner needs through regular and ad hoc feedback sessions including using surveys, class consultations on the quality of teaching and surveys on skills acquired. These responses were mapped to graduate profile outcomes (GPOs) and fed into the programme review process. One TEO holds a weekly lunch hui with the learners and collates learner feedback through this hui. This was because the TEO realised that their learners preferred to give verbal and face to face feedback. Another TEO ran a daily breakfast session for learners. These kinds of informal sessions are useful as an information gathering process to feed into programme review.
• Four of the TEOs focussed on Te Ao Māori principles and pedagogies and showed evidence of strong cultural awareness and responsiveness by involving Māori and Pasifika stakeholders in the programme design, delivery and review processes. These TEOs have indicated a willingness to network and share best practice with TEOs wishing to incorporate mātauranga Māori pedagogies into teaching, learning and assessments.
• Some high performing TEOs hold advisory group meetings to review and monitor programmes. These advisory groups usually comprise of past graduates, referral providers, Iwi kaumātua, the local community representative, industry leaders and - in one case - a secondary school representative. These meetings are used as an avenue to reflect on internal and external stakeholder engagement, and to shape ongoing programme delivery.
• All of the TEOs monitored have cultural safety at the forefront of delivery and have regular kōrero with tutors, management staff, Māori and Pasifika representatives and whānau. The kōrero also feed into annual self-assessment reports and quality assurance policy documents. All the TEOs provided evidence of positive comments from stakeholders on inclusion of tikanga practices in learning activities.

Currency and content of programme

Areas of good practice:

• Programmes are structured to enable learners to develop soft skills such as self-management, problem solving, communication, literacy and numeracy skills.
• Assessment materials are designed to meet learner needs. For example, one TEO noticed a considerable increase in assessment completions by moving to more practical learning activities in the afternoons such as music, sports and carving.
• Most TEOs make minor changes to learning activities as they are required. For example, one TEO changed the structure of the learning activities and integrated the groups of unit standards delivered so learners have several opportunities over the course of the programme to achieve the required level of competence.
• The use of guest speakers, tutor-led learning, work placements, off site trips, and marae-based learning to add variety and focus on specific areas of learning. These also increased learner engagement. One TEO offered learners options to complete an internship abroad as part of the programme.
• Project based learning promoting real life and real time experiences, leading to a mixed delivery method were also very effective – for example a bake sale project spanning several weeks incorporating relevant unit standards and GPOs. This also included the use of short lessons and learner-designed/led projects, strong engagement with wider whānau (encouraged to participate and support learning activities) and internal work placement within the TEO for some learners. This promotes a variety of teaching and learning methods, and there was strong evidence to show an increase in learner engagement, progression, retention and re-engagement.
• All TEOs have appropriately qualified staff with extensive industry and sector experience.
  • Four of the TEOs had staff and the senior management team actively involved in the community and local businesses, which translated into connections that inform the currency, content and relevance of the programmes.
  • Some tutors and managers from some of the TEOs monitored ran businesses where the learners undertook work placements as part of the course. This brought the learners into direct contact with industry experts with teaching and assessing experience, which underpins the ongoing evaluation of the currency and content of the programme.
  • Some TEOs supported staff to complete compulsory refresher courses in specialist fields to maintain certification.
  • There is evidence of ongoing professional development for staff at all TEOs monitored, and a good example from a TEO that organises a six-weekly community of practice session where staff share best practice, reflect on teaching methods and practice, discuss consistency of delivery and outcomes for learners and focus on current industry practice.
  • Some of the TEOs have tutors involved in the design of the programme and have scheduled peer observations.
• There is good evidence of efficient pre-and-post assessment moderation through Standard Setting Bodies, NZQA and external moderators. A couple of TEOs utilise the cluster moderation method. There was evidence this promoted consistency of delivery across multiples sites and promotes sharing best practice.
• Most TEOs have robust communication and contact with the relevant industries, and adapted the qualifications based on industry feedback to better suit the needs of relevant employers.

Quality of outcomes for learners and other stakeholders

Completion rates on foundation skills courses can be historically low due to learner demographics. However, of the monitored TEOs, ten have completion rates above 50%. Three of those ten have 100% completion and progression rates.

Areas of good practice:

• The TEOs with high completion rates had specific pathways for learners to progress towards employment or further education. For example, learners have to complete Level 1/Level 2 to progress to Level 2/Level 3 of the course. Three TEOs had a work placement element in the programme, which was between three to six months long. This provides learners with real world experience and an ongoing inside knowledge of industry demands, which can be mapped to the theoretical elements of the programme.
  • These TEOs had tutors who currently practiced in industry, thereby filling a dual role of supporting learners in the classroom and in the workplace.
  • These TEOs provided extensive pastoral support after graduation, with one TEO providing support for three months following graduation. This means the graduates had consistent continuity of care and support in the programmes and employment they had progressed into.
• All of the TEOs provide extensive pastoral care for learners, especially for those enrolled in Level 1 and 2 programmes. Examples include keeping whānau in the loop, breakfast clubs, support to access youth and community services, student grants, informal capturing of soft skills gained, supporting learners to achieve extra milestones such as getting driving licences, HGV licences, accessing accommodation, or providing laptops, iPads, internet access and phones to promote learning and engagement.
• TEOs with flexible enrolment processes had a higher number of learners progressing and completing the programme.
• Learner surveys from all TEOs show learners found learning and assessment activities diverse, engaging and challenging enough to promote an active ongoing interest in the programme.

Areas that need strengthening:

The monitoring activity highlighted some areas where programme review processes could be strengthened. For example:

• Some TEOs do not adequately capture information on the employers and industries which learners progress to.
• Some TEOs do not proactively collect and analyse progression information and feedback from employers.
• There is a lack of adequate data and evidence on how informal internal review data is effectively processed to feed into a formal programme review. For example:
  • There is a lack of adequate responses to graduate and graduate employer surveys, including a lack of input from external stakeholders from a majority of the TEOS.
  • Some TEOs are not adequately capturing narratives from learners and whānau on the value-added soft skills gained, including greater engagement with services, improved personal or family life.
  • Some TEOs have very minimal evidence of how staff feedback on the quality of programme design and delivery, assessments, moderation, resources and learner achievement feeds into programme review.
  • Some TEOs did not have documented action plans or evaluations of completed action plans and a reflection on how these have influenced the quality, performance and outcomes of the programme.
• Some TEOs lacked evidence of specific engagement with relevant community stakeholders for learner groups that identify as Māori and Pasifika.
• Some TEOs did not have any industry and employer engagement and input into programme review processes. Stakeholder input from whānau, employers, community and support agencies and TEOs will provide valuable information as to programme currency, content and consistency of delivery.
• Some TEOs do not capture a documented timeline of programme review activities. An effective programme review process should document the identifying of issues, who is accountable for managing the change and reporting back on completed review activities. This means the action plan processes of those TEOs are not robust in effecting change. It is important to document informal review activities such as verbal evidence from learners, and to use this evidence to inform programme review improvements.
• There was limited evidence provided from some of the multi-site TEOs on specific regional stakeholder engagement or input into the development, currency or updating of the programme.

Recommendations

Overall, NZQA found the outcome of this programme review monitoring activity reassuring, in that most TEOs are implementing effective programme reviews and continuing to meet requirements for maintaining approval and accreditation. NZQA recommends that TEOs:

• strengthen cultural awareness and responsiveness by designing programme review processes and policies that identify, analyse and address the needs of Māori and Pasifika learners.
• continuously review stakeholder consultation practices and ensure that programme evaluation policies reflect what occurs in practice.
• collect and analyse data on the consistency of programme delivery, currency and content across all delivery sites to ensure the programme meets the needs of the local/regional community.
• systematically collect and analyse feedback from organisations (TEOs and employers) to which learners progress. This provides evidence that progression pathways are appropriate for learner needs.
• develop strategies for increasing learner completion rates.
• consider using success stories/videos from employers/learners/families as a motivational tool for future cohorts.
• capture narratives from graduates on how the programme added value to their professional journey, particularly for programmes with a large practical element.