Monitoring of Level 7 diploma programmes: 2018 summary

The purpose of this summary is to:

• review the results of the 2018 monitoring of programmes leading to Level 7 diplomas
• outline the key findings
• provide guidance to ensure programmes continue to meet the programme approval and accreditation rules.

Programme approval and accreditation rules

This programme monitoring summary is primarily for:

• tutors and assessors
• programme developers and programme leaders
• academic and quality managers.

At the end of the report there are top tips for these three groups.

Download a pdf version of the report [PDF, 217 KB]

Monitoring activities

In 2018, NZQA monitored 16 out of 109 approved level 7 diploma programmes. These diplomas were in the following fields:

• Business
• Computing/Information Technology
• Digital Marketing
• Engineering
• Healthcare Management
• Hospitality Management.

NZQA’s risk-based approach

NZQA takes a risk-based approach to determining the type of monitoring activity.

Risk factors include but are not limited to:

• the number of student enrolments
• programmes with high numbers of international students
• NZQA monitoring and moderation history
• other intelligence from within NZQA or external agencies.

For the 2018 monitoring, all monitoring activities included a desk evaluation of programme documentation supplied by Tertiary education organisations (TEOs) and moderation of a sample of assessed learner work. 12 of the 16 monitoring activities also included a site visit to the provider to interview staff and learners.

The outcomes of NZQA’s monitoring activities

NZQA completed 16 monitoring activities and found that:

• one programme met programme criteria overall
• a further eight programmes met some programme criteria with some areas requiring remedial actions to meet programme approval and accreditation criteria
• the remaining seven programmes did not meet programme criteria overall and required significant remedial action or action leading to statutory intervention in some cases.

Applied and practical programmes

Monitoring found that programmes that were predominantly applied and practical rather than theoretical provided the best outcomes for learners and for employers. Graduate destination data showed this.

In particular, assessments involving students applying knowledge in real industry contexts allowed learners to fully demonstrate that they met the programme learning outcomes. These programmes also tended to have active reciprocal links with industry.

Common issues

The most common issues which prevented programmes meeting programme criteria included poor assessment and moderation practice and a lack of robust programme review.

The rest of this report explains the monitoring findings in more detail.

Assessment and moderation

This section of the monitoring report addresses:

• assessment methodology and procedures
• authenticity
• moderation systems and processes
• the results of NZQA’s moderation of a sample of assessed learner work.

Approved methods of assessment

Most level 7 programmes that NZQA has monitored use an achievement/grade-based marking system. NZQA will approve programmes with assessments that are competency/standards-based or achievement/grade-based.

There must be clear guidance for the learner and assessor on the level of performance required to meet each learning outcome.

Expectations for achievement/grade-based assessment

For achievement/grade-based assessment, NZQA moderators expect to see detailed criteria for each grade. The criteria for the minimum passing grade (typically a ‘C-‘or 50 per cent) must align with achieving the learning outcome(s) within that assessment(s).

Grades higher than a ‘C’ show work over and above what is needed to achieve each learning outcome.

Guidance on assessments for multiple learning outcomes

If the assessment task covers multiple learning outcomes, the assessor must ensure that the learner evidence meets the minimum passing criteria for each learning outcome.

For instance, if an assessment covers three learning outcomes, a learner might do exceptionally well on two learning outcomes, achieve a mark of more than 50 per cent, but has not achieved the third learning outcome. This is not appropriate as students should not pass if they have not provided sufficient evidence for each of the learning outcomes being assessed.

Types of assessments used

In the level 7 programmes NZQA monitored, assessment was comprised of two or three assessment tools for each programme component/module.

Most of these assessments were open book or written assignments, but there were occasional closed book tests or exams.

Presentations sometimes contributed to the final grade but more often they were part of a group assessment.

Group assessments

Group assessments usually accounted for less than half of the final grade of a course/programme module.

In the programmes NZQA monitored, many assessments relied on the group producing a shared product such as report or a presentation. It is difficult to assess individual achievement of the learning outcomes in this context.

Three programmes used practical projects to assess learners. Different group members were allocated tasks which meant they demonstrated only some of the learning outcomes.

Research and associated ethics

Most of the programmes requiring research activities involving people have an established process for applying for ethical approval of their research proposals.

Authenticity

All TEOs delivering level 7 diplomas recognise that ensuring the authenticity of learner work is critical to the quality assurance process. All TEOs monitored had policies and established practices for detecting plagiarism and the increasing use of synonym-replacement software. TEOs see tutors as the best defence against plagiarism as they know a learner’s strengths in their subject. Tutors also monitor a learner’s participation and performance in class activities.

Many programmes rely on similarity comparison software to help the assessor detect plagiarism. However, ensuring authenticity of learner work is more than just identifying plagiarism. A TEO’s policies and processes must include prevention as well as detection of academic dishonesty. None of the TEOs NZQA monitored had policies that explicitly referred to preventing and detecting contract cheating.

Academic integrity and AI

Conditions for assessment, resubmission and reassessment

NZQA also considered conditions of assessment, including resubmission and reassessment, as part of the programme monitoring.

Issues raised in this area included:

• Group assessments that do not allow the individual to meet all of the learning outcomes independently.
• Too much time to complete the resubmission or reassessment.
• Multiple opportunities for resubmission, particularly when the first attempt was minimal.
• Giving learners back the same examination script to complete a resubmission for a closed book assessment.

Moderation systems and processes

TEOs usually had policies and processes for programme moderation, but aspects were sometimes not operating during the monitoring or were not applied effectively.

These aspects included:

• No schedule for moderation.
• Some assessments being delivered without going through a pre-assessment moderation process.
• Moderation reports with limited feedback or that made no reference to the learning outcomes or effective assessment design.
• Moderation being carried out by non-subject specialists.
• No or limited post-assessment moderation being carried out.
• Moderation reports not filled out completely (such as names, signatures, dates).
• No action plans implemented and/or monitored as a result of moderation feedback.

The best moderation systems ensured that:

• assessment tools provided learners with the opportunity to meet all learning outcomes
• TEOs gather feedback on the clarity of instructions and the appropriate level of questions
• marking guidance was clear
• action plans arising from moderation reports were sufficiently detailed and monitored
• relevant staff received professional development.

NZQA moderation

NZQA conducted moderation of a sample of marked learner work from each of the 16 programmes monitored. The purpose was to verify if learners deemed to have passed had supplied sufficient evidence to meet the learning outcomes being assessed.

Rates for agreement between the NZQA moderator and the TEO assessor ranged from 0 to 91 per cent for the programmes monitored with an average of 38 per cent.

The main reasons for not verifying assessor decisions include:

• Insufficient learner evidence to meet the learning outcomes.
• Learner work not at the required level. Examples of this include providing a description when the learning outcomes require critical analysis or providing a response that does not refer to the theoretical underpinning expected at level 7.
• Lack of academic writing conventions such as referencing.
• Poor English that inhibits the learner from demonstrating their understanding or meeting the requirements of the New Zealand industry context.

Poor assessment practice

Other issues of poor assessment practice noted by NZQA moderators include:

• Assessment tasks that do not allow the learner to meet the approved learning outcomes.
• Learners passing an assessment and/or module despite not meeting the minimum passing criteria for each learning outcome.
• Marking guidance that is not detailed enough to allow for consistent marking.
• Over-generous marking.
• Little constructive feedback by the assessor.
• Potential authenticity issues such as heavy reliance on published material or copying another learner’s work.
• Reliance on publicly available information in a subject that requires much more detailed information – for example from inside the business.
• Undetected use of synonym replacement software, which in some cases made the learner’s work meaningless.

Good assessment practice

One example of good assessment practice that NZQA noted was unifying two assessments by requiring the learner to extend their chosen context from the first assessment into the second assessment.

The focus of the learning outcomes was on the development from introductory to more advanced skills. Using the same context meant that the learner could focus their efforts where it was most needed rather than researching another context.

Another good practice example involved the TEO partnering with a business producing an automobile prototype. The business allowed learners to base their final/capstone projects on one element of the prototype.

Learners were assessed in a real-world context and designed solutions to match a design brief. The prototype was a large project, allowing learners to meet the learning outcomes individually, even while working as part of a group.

Programme structure and delivery

This section of the monitoring report addresses:

• learning outcomes
• delivery methods
• programme length
• learning hours.

When delivering an applied or highly practical programme it is good practice to give learners the opportunity to actually apply those skills. Good examples of this in the programmes NZQA monitored include practical “lab” sessions, projects, industry simulation activities, and field trips.

Timetabled hours in class may include lectures, tutorials or practicals, but these should also consider learner needs. Two TEOs delivered programmes in a compressed format and several more programmes had long blocks of class time, with or without formal breaks. Learners commented that they struggled to stay motivated and engaged during these long sessions.

Most of the programmes effectively supported learners with structured activities and set readings to complete outside of class time. This was actively monitored by follow-up activities in class or through online learning management systems.

Changes to programmes

Changes to programmes are a natural outcome of programme review: i.e. reviewing the programme learning outcomes. Over time learning outcomes may be refined, combined or not needed.

NZQA must be notified of any changes to approved learning outcomes to ensure the programme still allows the learner to meet the graduate profile.

Six of the programmes monitored contained unapproved changes to the learning outcomes or programme structure.

Resources

This section of the monitoring report addresses:

• academic staffing and professional development
• teaching or learning facilities
• educational resources
• student support.

Issues with academic staff

The biggest area of concern was the academic staffing for half of the level 7 diploma programmes NZQA monitored. All teaching and assessment for level 7 programmes should be conducted by sufficiently experienced and qualified staff.

This means:

• having a qualification at level 8 or above in the subject they are teaching
• having or be in the process of completing an adult education qualification
• experience working in the relevant industry.

A significant number of TEOs employed staff without the above qualifications and staff were often required to teach outside of the subject area of their level 8 qualification: for example, an accounting specialist teaching marketing.

The depth of knowledge and level of engagement required at level 7 means teaching staff must have a specialised qualification. Otherwise they cannot assist learners to meet the learning outcomes of level 7 diplomas.

Providing professional development

It is reasonable that staff should expect support and encouragement from their employers to develop their capabilities as teachers.

Staff who NZQA interviewed spoke highly of their employers’ investment in professional development. Some employers provide in-house or external professional development programmes and/or courses. Others also pay for teachers to complete adult education qualifications.

This support could be expanded to help teachers develop links with industry to draw on current trends and practice and gain more practical/work experience. This was often a gap in staff expertise.

Providing student support

Resourcing of student support was consistently good across all programmes NZQA monitored.

Students were supported before starting on the programme, took part in an orientation programme for a few days or week at the start of the programme and were given ongoing pastoral support.

Some TEOs also offered academic writing, English language support and career guidance to students outside of the programme requirements.

Providing appropriate facilities and resources

Teaching and learning facilities and education resources were also appropriate for the programmes NZQA monitored.

NZQA notes TEOs are increasingly using online learning management systems (LMS) such as Moodle to facilitate learning in and out of class. An LMS can be used to:

• share learning resources
• facilitate learning activities (such as quizzes and discussion forums)
• track student participation and engagement.

NZQA also notes that campus libraries are decreasing in favour of resources provided through community or shared university libraries.

TEOs are giving learners access to peer-reviewed journal articles through these facilities or independently for the programme. This is a vital aspect of study at this level.