Monitoring of tertiary education organisations’ internal moderation systems: 2020 to 2024 summary

Tukanga arotake | The monitoring process

For the internal moderation monitoring, NZQA gathered and evaluated evidence in relation to a TEO’s internal moderation system and sought to answer the following key evaluative question (KEQ):

The internal moderation monitoring process involves the review of the following:

• moderation policies and processes
• internal and any external pre- and post-assessment moderation activities
• assessment materials and assessor decisions
• examination and evaluation of learner samples
• teaching, assessor and moderation staff credentials
• actions arising from moderation.

The selection of programmes for monitoring is a risk-based and data-driven approach. Areas of risk identified include programmes:

• with high domestic and/or international enrolment numbers
• with significant practical components, e.g., practicum, work placement in clinics or kitchens
• delivered by TEOs previously identified as non-compliant
• delivered by TEOs with new owner(s)
• delivered in part by subcontractors or employers
• delivered by TEOs outside their previous areas of specialisation.

Programmes at Levels 1 to 6 were selected for monitoring from different subject areas, such as:

• Religious Studies
• Sport and Recreation
• Community Client Care (Health and Wellbeing)
• Tourism Studies
• Hospitality
• Nannying and Early Childhood Education
• Horticulture
• Beekeeping
• Personal Services
• Massage Therapy
• Naturopathy and Homeopathy
• Graphic Arts and Design Studies
• Food and Beverage Service
• Security
• English Language.

During the monitoring, NZQA requests a set of documents from each TEO for evaluation, followed by interviews. 

The monitoring sought to provide assurance that TEOs are continuing to meet requirements for maintaining programme approval and accreditation, including:

• the effectiveness of the TEO’s ongoing policies and practices for moderation of assessment materials and assessor decisions.
• the capability and capacity of the TEO to ensure that the assessment materials and decisions are fair, valid, consistent and appropriate for the level, given the stated learning outcomes.
• that TEOs are self-identifying issues through their moderation processes and are actively using those to improve programme quality.
• the currency, content and level of the programme.

Ngā kitenga | Monitoring findings

Areas of good practice

This section highlights areas of good, effective or innovative moderation policies, practices and staff capability.   

Examples of good practice are:

• clearly articulated policies for internal and external pre- and post-assessment moderation, which outlined how the processes will occur, the frequency with which they should occur and the personnel responsible for each step.   

• policies requiring pre-assessment moderation of newly developed or modified assessment materials, including those acquired from commercial sources or external partners.   

• policies and processes ensuring that any changes required as a result of pre-assessment moderation were made and approved by the moderator and/or quality assurance manager prior to delivery to the next cohort.   

• a clearly articulated process involving regular review of moderation findings and evidence of discussions or formal meetings held to examine and refine assessment materials, assessor decisions and the associated resources.   

• using the moderation process to inform continuous improvement in their assessment and moderation practices and to build staff capability.   
• detailed schedules for internal and external pre- and post-assessment moderation across cohorts and, where applicable, across delivery sites. The more comprehensive schedules included the dates for moderation, names of assessors and moderators involved, and clearly identified the courses/programmes/modules/unit standards to be moderated. Many schedules spread moderation across more than one year, and most ensured that moderation covered all papers/courses/modules/unit standards within a programme at least once in a three-year period.  
• additional moderation of assessor decisions based on the skills and experience of the tutor/assessor involved.  Where a tutor/assessor was new, unfamiliar with the programme, or had been previously found to be inconsistent in their judgements, the TEO would choose to moderate their assessment decisions more frequently. This moderation would occur before the learners’ marks are finalised and the release of the marked assessments to mitigate any risk of poor decisions.   
• well-developed pre-assessment moderation forms, which directed the moderator to consider whether assessment tasks gave learners opportunities to fulfil the requirements of each approved learning outcome at the appropriate level. In examples of best practice identified by NZQA, moderators were also required to comment on the clarity of instructions, the validity of the assessment method, the assessment conditions and whether the marking guidance ensured that learners provided sufficient evidence to meet all the stated learning outcomes.  
• well-developed post-assessment moderation forms, which directed the moderator to consider whether assessment tasks had given learners opportunities to meet the stated learning outcomes at the appropriate level. The most comprehensive forms also directed moderators to consider whether learners have provided sufficient evidence at the appropriate level to meet all the requirements of each of the learning outcomes. The forms included space for the moderator to comment on the quality of assessor decisions, any need for staff development, and recommendations for amendments to the assessment materials and/or the assessment conditions. In a few instances, there was also a section listing follow-up actions required and a sign-off from the moderator that those steps had been actioned effectively.   
• an effective system of tracking and recording the results of internal and external pre- and post-assessment moderation.  
• TEOs clearly stipulated the minimum qualifications and/or experience required to become an internal moderator. Where the programmes included unit standards, these requirements reflected the relevant Consent and Moderation Requirements (CMR) requirements and the TEO’s own Quality Management policy.  

• TEOs were proactively encouraging staff development in assessment and moderation practice by supporting them to gain unit standards such as 11551 Moderate assessment and 4098 Use standards to assess candidate performance and related adult teaching qualifications and/or providing their internal staff development workshops. 

Common assessment and moderation issues

This section highlights the most common assessment and moderation issues NZQA found during its monitoring of TEO internal moderation systems. We share them here to help TEOs avoid similar issues in future and to support effective assessment and moderation practice.

Assessment design and method

Most of the issues identified below stemmed from poor choice of assessment method and poor assessment design which greatly impact learner achievement of the programme learning outcomes. These include:

• Assessment tasks do not provide learners with opportunities to meet all the requirements of the programme learning outcomes.
• Changes in the wording of the learning outcomes from those approved by NZQA. Many TEOs had altered words within the learning outcomes or had omitted significant parts of the learning outcomes on course outline documents, in assessment materials and in marking guidance. For example:

• • changing ‘evaluate’ to ‘explain’ decreases the level of analysis and cognitive demand required of the learner and does not direct them to form an opinion and give reasons as required.
  • omitting the required context, like ‘in a healthcare setting in Aotearoa’, does not ensure that learners provide evidence of their specialised knowledge within that environment.
• Misreading and/or misinterpretation of learning outcomes. NZQA stresses that where a plural form is used, e.g. children, whānau, schools, concepts, the learning outcome requires at least two examples of that person/item/idea/occurrence. Similarly, if the learning outcome requires ‘a range of’ or ‘a variety of’, it is expected that at least three examples will be used and that there will be some variation/differences between those examples.   
• Changing the approved assessment methods or methodology without advising NZQA or seeking NZQA approval (for Type 2 change). For example, changing a practical project to an essay may disadvantage learners who are more kinaesthetic. It can also lead to an imbalance between theory and application of skills and knowledge and may no longer give learners opportunities to meet the approved learning outcome and the full extent of the graduate profile outcome(s) to which the learning outcomes relate. For example, if the learning outcome is ‘demonstrate effective communication skills’, but the task is changed from a collaborative project to a written essay, the learners can no longer provide evidence of the range of interpersonal, non-verbal, aural, and verbal skills that are required.
• Use of assessment methods like multi-choice questions or short answer quizzes which may prevent or limit learners to show the depth of their understanding or the extent of their skills and/or cognitive ability at the appropriate NZQCF level.
• Not providing opportunities for practical demonstration of skills/knowledge/attitudes where this is required by the learning outcome. For example, ‘apply engagement skills’ or ‘demonstrate safe-handling practices’ require that the learner actively shows their ability to produce the evidence required at the appropriate level. Self-reflection or reporting what they did does not meet the requirements. Learners should be observed completing these practical tasks, and there should be comments from the assessor/verifier, which are specific to the learner and describe in detail what the learner said and/or did that show that the learner has met the learning outcome(s). Audio-visual recordings can also be used to provide this supplementary evidence, but NZQA is aware that this is not always possible.
• Incorrect use of photographs or screenshots. Where a learner is being assessed on their ability to undertake a practical task or a process that involves steps or stages, each of these milestones should be captured. Supplying only one or two photographs or screenshots of the finished products does not provide sufficient evidence that the learner independently completed the process. Assessment tasks should give learners precise instructions about what shots are required and how many. Marking guidance should also detail these requirements.
• Incorrect use of open-book assessment conditions. Open-book assessments are not appropriate where the learner is required to provide clear evidence of their personal, independent understanding of an idea or concept or where they need to show evidence that they have learned and retained certain concepts or key information. If the open-book conditions simply allow the learner to find information and transfer it to the answer sheet or to select the correct answer from a multi-choice or short answer array, then the assessment conditions may prevent the learner from meeting the requirements of the learning outcomes.
• Incorrect use of group assessment. If a learner is being assessed in a group context, there must be clear opportunities for the learner to produce evidence of their individual ability to meet all the requirements of all the learning outcomes within that context. The group cannot be assigned a blanket mark for a combined outcome and cannot do pieces of a task if that means they do not have an opportunity to meet all the learning outcome requirements by only doing that piece. For example, doing the graphics for a presentation does not provide evidence that the learner can speak effectively in public. Writing an introduction for a group report does not show that the learner can independently produce logical conclusions and recommendations.

Programme/course completion statements and marking guidance

Moderation should identify whether the marking guidance directs the assessor to ensure that the learner evidence has met all the learning outcomes at the appropriate NZQCF level. Moderation should also identify where assessors have awarded learners passing marks or grades without meeting all the learning outcome requirements.  Several common issues were identified:

• Overall programme/course pass requirements and/or marking guidance that does not specify that learners must meet all the outcomes at the appropriate level for that programme.
• Overall programme/course pass requirements based solely on accumulated marks where the minimum grade (C-) or ‘Pass’ (50% mark) does not align with the minimum learner evidence/performance required to meet the learning outcomes.
• Marking guidance that does not give clear qualitative and quantitative descriptions of the minimum evidence required to meet all the learning outcome requirements.
• Use of marking rubrics with broad bands, for example, 0-5 or 6-10, where there is no indication of how individual marks are allocated and how assessors are to decide which end of the band to allocate.
• Use of marking rubrics with subjective or imprecise wording, which can lead to unfair and inconsistent marking. For example, ‘mostly measured’, ‘partly measured’, and ‘slight significant impact’.
• Use of marking rubrics where a marking schedule would be more appropriate as it guides the assessor to check evidence for specific details and/or allocate marks to specific points.
• Marking guidance that is focused on the task requirements rather than aligning fully to the learning outcome requirements. In this case, assessors may have followed the marking guidance accurately but may not have their decisions verified by moderation as the learner evidence does not meet the learning outcomes.
• Marking guidance that does not require the learner to include the correct number and/or range of examples required to meet the learning outcome. 

Assessor decisions

In many cases, inaccurate and/or inconsistent assessor decisions were directly linked to poor assessment design and marking guidance. If the assessment task and/or the evidence requirements and marking guidance were not fully aligned to the learning outcomes, the assessor decision was also likely to be incorrect.

Other issues with assessor decisions included:

• Use of check-marking which was seen by the TEO as post-assessment moderation. Check-marking is the practice of ensuring the assessor has followed the marking guidance accurately. Moderation is ensuring that the learner evidence has met the requirements of the learning outcome/unit standard at the appropriate level. They are not synonymous.
• Over-generous marking, where learners were awarded higher marks than allocated in the marking guidance or awarded a grade higher than their evidence warranted.
• Excessive assessor feedback, including editing, which provides too much detail and thus undermines the learners’ ability to produce evidence that they can independently meet the approved learning outcomes. 
• Inaccurate calculation of marks and or total marks.
• No indication of where marks have been awarded and/or no feedback to learners.
• No indication of where a learner sits within rubric bands and/or inconsistent allocation of the higher or lower mark in the band which may skew the learner results when compared to another assessor.
• Use of checklists without adequate commentary to explain how the learner evidence/performance has met the requirements of learning outcome/performance criteria.
• Inconsistent marking of referencing requirements and in-text citations, even where penalties or marks are included in the marking guidance.
• Assessors/verifiers/moderators failing to complete all the required fields in the documents required as part of assessment and moderation records. For example, not signing and dating attestation sheets. 
• Failure to record what learners said for oral resits or when questioned for clarification.

Lack of detailed assessment and moderation policy

NZQA identified that, in many cases, policies required improvement to direct assessors and moderators more effectively and to ensure that moderator findings and recommendations were followed through.

Common issues with assessment and moderation policies were:

• Insufficient detail about steps within the moderation process and who is accountable at each step. For example, a policy that does not specify how many learner samples will be selected for post-assessment moderation and what criteria, if any, will be used to make that selection.

• A lack of definition of terms like resit/resubmission/RFE (request for further evidence) and does not make a clear distinction between them in terms of requirements for the learners and assessors.
• Plagiarism is not clearly defined and/or does not detail how it might be prevented and detected and what the consequences to the learner might be.
• The use of generative Artificial Intelligence (AI) is not covered. An authenticity policy should describe when the use of AI is acceptable and under what conditions.
• Qualifications and/or experience in assessment and moderation are not clearly defined for internal and external moderators.

Moderation process

For moderation to be fully effective, all moderator findings should be considered, and modifications made where appropriate. The decisions leading to those modifications should also be recorded as evidence of using moderation as a tool for continuous improvement. However, the following issues were identified:

• Many TEOs did not monitor the actions/recommendations arising from pre- and post-assessment moderation. It was not clear whether the moderator’s comments had been actioned and whether the moderator had signed off the changes prior to delivery to the next cohort. This meant that it was not clear how the TEO had used moderation to improve their delivery and assessment practice.
• Some TEOs did not have an adequate system of tracking that pre- and/or post-assessment moderation had occurred on a regular basis. In some cases, this meant that some courses were moderated regularly, while other parts of the programme were not.
• Despite rules requiring retention of learner evidence, many TEOs were unable to provide full sets of learner samples, and several lacked evidence of the practical components of learner work, including presentations and projects.
• Although several TEOs had pre- and/or post-assessment moderation conducted by another standard-setting body or an external entity (e.g., CAA, PADI), they were unable to produce moderation reports from those activities and could only report the outcome without the associated paperwork as verification.
• NZQA found that many providers had pre- and post-assessment moderation forms, which did not direct moderators to consider critical elements of the moderation process. These included:  
  • Pre-assessment and/or post-assessment forms that do not direct the moderator to consider whether the assessment task(s) gives learners opportunities to meet each of the approved learning outcomes. 
  • Pre-assessment and/or post-assessment forms that do not direct the moderator to consider if the marking guidance aligns with each of the learning outcomes and/or if it provides objective, clear qualitative and quantitative descriptions of the evidence required.

Conditions for assessment, resubmission and reassessment 

NZQA also considered assessment conditions, including resubmission and reassessment, as part of the internal moderation monitoring. Pre- and post-assessment moderators must consider if the assessment conditions are appropriate to the task design and learning outcome requirements and provide assurance of fair and consistent assessment outcomes.

Issues included:

• Too much time to complete the resubmission or reassessment.
• Lack of clear policy on the number of attempts permitted and/or assessors not adhering to the policy.
• Multiple opportunities for resubmission, especially when the first attempt was minimal.

Learners receiving back the same examination/test script to complete a resubmission for a test or assessment.

Authenticity

All TEOs monitored recognised that ensuring the authenticity of learner work is critical to the quality assurance process. Most had policies and established practices for detecting plagiarism and the increasing use of synonym-replacement software. TEOs generally see tutors as the best detector and defence against plagiarism as they know a learner’s strengths in their subject. Tutors also monitor a learner’s participation and performance in class activities.

However, issues with authenticity included:

• Inconsistent and/or inaccurate application of policy, including the penalties for detected plagiarism.
• Lack of policies and processes aimed at preventing plagiarism and academic dishonesty. Few of the TEOs monitored had policies that explicitly referred to preventing and detecting contract cheating.
• Using the same unaltered assessment materials for several cohorts or successive years, which creates the risk that they will be passed on to subsequent students.
• Focus on plagiarism associated with written resources and a lack of policies regarding the use of visual images and code sourced online.
• Conditions of assessment which may not be appropriate, for example, open book or unrestricted access to the internet do not ensure academic integrity is maintained.

Staff capability

As part of Rule 7.1.1. NZQA considered staff capability and capacity to ensure assessment materials and decisions are fair, valid, consistent and appropriate for the level given the stated learning outcomes. Findings included:

• Many TEOs had staff who were well-qualified in their specialism but lacked qualifications and/or experience in assessment design and moderation. Some did not have adult teaching qualifications and/or hold standards 11551 Moderate assessments, 11552 Design and evaluate assessment materials and/or 4098 Use standards to assess candidate performance or equivalent but were still responsible for either assessment design or moderation or both. Flaws in assessment design or assessor practice may reflect the lack of adequate training and/or understanding of the importance of the moderation processes in maintaining fair, valid and consistent assessment.
• Some TEOs were delivering and assessing against unit standards with staff who did not meet the requirements of the associated CMR.