Monitoring of New Zealand Diploma in Business (Level 5 and 6): 2018 summary

In 2018 NZQA began monitoring the programmes leading to New Zealand Diplomas in Business (Levels 5 and 6) [Refs: 2459 and 2460].

This summary:

• outlines the key findings
• provides guidance to ensure programmes continue to meet programme approval and accreditation rules.

This summary of programme monitoring is mainly for:

• tutors and assessors
• programme developers and programme leaders
• academic and quality managers.

There are also top tips for these three groups.

Download this report as a PDF [PDF, 216 KB]

Programme approval and accreditation rules

Monitoring activities

In 2018, NZQA monitored:

• 21 out of 42 approved Level 5 programmes leading to Level 5 New Zealand Diploma in Business [Ref: 2459], and
• 11 out of 30 approved Level 6 programmes leading to Level 6 New Zealand Diploma in Business [Ref: 2460].

The 32 programmes NZQA monitored were delivered by 21 tertiary education organisations (TEOs).

For the 2018 monitoring, NZQA used the following three types of monitoring:

• Moderation only: NZQA moderated a sample of assessed learner work.
• Desk evaluation: NZQA evaluated the programme documentation that TEOs supplied, including NZQA moderation.
• Visit: NZQA made a site visit, including interviews with relevant staff members and learners, as well as a desk evaluation.

The 32 programmes were monitored as below:

The outcomes of NZQA’s monitoring activities

NZQA found that many of the programmes did not meet the criteria for programme approval and accreditation.

Of the 21 Level 5 programmes monitored:

• two programmes met programme criteria overall
• a further five programmes met some programme criteria with some areas requiring remedial actions to meet programme approval and accreditation criteria
• the remaining 14 programmes did not meet programme criteria overall and required significant remedial action. In some cases, NZQA made a statutory intervention.

Of the 11 Level 6 programmes monitored:

• no programme met programme criteria overall
• a further three programmes met some programme criteria with some areas requiring remedial actions to meet programme approval and accreditation criteria
• the remaining eight programmes did not meet programme criteria overall and required significant remedial action. In some cases, NZQA made a statutory intervention.

Common issues

The most common issues which prevented programmes meeting programme criteria included:

• poor assessment and moderation practice
• unapproved changes to the programmes
• a lack of robust programme review.

The rest of this report explains the monitoring findings in more detail.

Assessment and moderation

This section of the monitoring report addresses:

• assessment methodology and procedures
• authenticity
• moderation systems and processes
• the results of NZQA’s moderation of a sample of assessed learner work.

Assessment methodology and procedures

Most of the Level 5 and 6 business programmes that NZQA has monitored use an achievement/grade-based marking system. NZQA approves programmes with assessments that are competency/standards-based or achievement/grade-based.

Both the learner and the assessor must have clear guidance on the level of performance required to meet each learning outcome. Assessing against learning outcomes that map to the graduate profile is central to assuring the quality and integrity of programmes.

Expectations for achievement/grade-based assessment

For achievement/grade-based assessment, NZQA moderators expect to see detailed criteria for each grade. The criteria for the minimum passing grade (typically a ‘C-‘or 50 per cent) must align with achieving the learning outcome(s) within the assessment(s).

Grades higher than a ‘C’ show work over and above what is needed to achieve each learning outcome.

Guidance on assessments for multiple learning outcomes

If the assessment task covers multiple learning outcomes, the assessor must ensure that the learner evidence meets the minimum passing criteria for each learning outcome.

For instance, if an assessment covers three learning outcomes, a learner might do exceptionally well on two learning outcomes, achieve a mark of more than 50 per cent, but has not achieved the third learning outcome.

This is not appropriate, as learners should not pass if they have not provided sufficient evidence for each of the learning outcomes being assessed.

Assessment methods

In the Level 5 and 6 business programmes NZQA monitored, assessment often consisted of two to four assessment tools of varying weightings for each programme component/module.

Assessment methods varied across components between programmes. They included:

• written reports
• presentations
• New Zealand-based case-study analyses
• group-based assessments
• examinations and portfolio submissions.

In some cases, the assessment method used was not relevant to the learning outcomes assessed: for example, using an exam with multiple-choice questions when the learning outcome required explanation and implementation of a certain concept.

Using relevant assessment tools is critical to designing assessments that ensure learners meet certain learning outcomes.

Group work and group assessment

Every programme monitored included group assessments. Group assessment is different to group work. In group work, learners collaborate as part of the learning process, but they are assessed individually. In group assessment, learners are typically awarded the same assessment result for something they produced collectively.

Group assessment is a valid method of assessment for Business programmes. It is an overarching condition of these qualifications that assessment must be conducted in and for the context of a real or realistic business entity. Teamwork and collaboration skills are fundamental to working in a business at operational and strategic management level.

However, it is important to ensure that group assessment is only used where it makes sense to use it (i.e. where it is appropriate for the learning outcome being assessed). It is also important that all group members have contributed to the assessment task to a degree where the assessor is sure of everyone’s ability in relation to the learning outcomes. Overusing group assessment makes it difficult to provide this assurance.

In many of the programmes monitored, assessments relied on the groups producing a shared product such as a report or a presentation. It was difficult to assess individual achievement of the learning outcomes in this context. There was no evidence showing which member of the groups produced which part of those assessments. Group work rather than group assessment may be more appropriate in these cases.

Two programmes that NZQA monitored used individual tasks within a group-based assessment, which were measured independently to ensure that learners had met all learning outcomes assessed. Moderators considered this a useful method because it provides learners with the opportunity to collaborate and achieve the learning outcomes.

Recording group presentations and using individual assessment forms (e.g. individual tasks mapped against each learning outcome being assessed) were also useful ways of collecting evidence for a group-based assessment.

Overall, NZQA recommends that group assessment is used conservatively in programmes leading to the award of the Level 5 and 6 Business qualifications. Instead, NZQA encourages TEOs to use a variety of assessment methods. These methods should be designed to reflect the learning outcomes and the context and requirements of a New Zealand business.

Conditions for assessment, resubmission and reassessment

NZQA also considered conditions of assessment, including resubmission and reassessment, as part of the programme monitoring. NZQA identified some issues which hindered learners from demonstrating their independent learning and may have prevented them from developing the capability to repeat their performance in an unfamiliar context.

Issues raised in this area included:

• learners completing their assessments in class with excessive guidance from tutors
• learners submitting drafts multiple times before submitting their assessments
• TEOs allowing multiple opportunities for resubmission, particularly when learners received excessive feedback on the first attempt.

Authenticity

All TEOs delivering the Level 5 and 6 business programmes recognise that ensuring that learner work is authentic is critical to the quality assurance process. All TEOs monitored had policies and established practices for detecting plagiarism and the increasing use of synonym-replacement/text-rewriting software. Most programmes relied on plagiarism detection software as the first line of screening, and each tutor then assessed the similarity and relevance of the content in the learners’ work.

Learners who NZQA interviewed at monitoring visits understood plagiarism well, but NZQA still identified potential authenticity issues in learners’ work the TEO submitted for moderation. For example, NZQA moderators were unable to verify whether the submitted work was the learners’ own materials due to the overuse of unreferenced citations in their work.

NZQA also notes that ensuring learner work is authentic is more than just identifying plagiarism at the time learners submitted their work. For example, the issue of learners using synonym-replacement/text-rewriting software was specifically raised for three Level 5 and three Level 6 business programmes. Due to the careless use of that type of software, learner work lost meaning and became hard to comprehend.

A TEO’s policies and processes must include prevention, as well as detection, of a wide range of academic dishonesty.

Academic integrity and AI

Moderation systems and processes

All but two TEOs had relevant policies and processes for their internal and external moderation. However, NZQA found that the actual practice was often not effective.

Issues found in this area included:

• TEO’s internal moderators failing to identify critical issues in assessments or assessor judgements, particularly issues that NZQA moderators identified (e.g. learning outcomes not being achieved by learners) 
• TEO’s internal moderation reports containing limited feedback, or no reference to the learning outcomes or effective assessment design
• TEOs not addressing issues identified by moderation
• no follow-up moderation in response to assessments failing pre-assessment moderation
• no evidence of TEOs conducting pre- or post-assessment moderation
• lack of clear oversight of moderation system by TEO management
• no plans of proposed actions to address issues identified by moderation. 

The best moderation systems ensured that:

• assessment tools allowed learners to meet all learning outcomes
• TEOs gathered feedback on the clarity of instructions and the appropriate level of assessment tasks
• there was clear guidance for marking assessments
• action plans arising from moderation reports were sufficiently detailed and monitored
• moderation activities were clearly scheduled in a regular cycle
• staff received professional development in assessment and moderation. 

Some TEOs had a dedicated moderation/quality assurance team or a specific staff member overseeing their moderation practice. These TEOs were able to provide a holistic picture of their assessment practice. NZQA found this was an effective way of managing moderation.

NZQA moderation

NZQA moderated a sample of marked learner work from each of the 20 Level 5 business programmes and 10 of the Level 6 business programmes NZQA monitored. For one Level 5 and one Level 6 programme, NZQA could not conduct valid moderation because monitoring identified that the TEO had made significant changes to the learning outcomes.

The purpose of NZQA’s moderation was to verify if learners the TEO had passed had supplied sufficient evidence to meet the learning outcomes being assessed.

Rates for agreement between the NZQA moderators and the TEO assessors ranged from 0 to 76 per cent for the Level 5 programmes monitored with an average of 21 per cent. Agreement rates ranged from 0 to 41 per cent for the Level 6 programmes monitored with an average of eight per cent.

The main reasons for not verifying assessor decisions include:

• insufficient evidence that learner had met learning outcomes
• learner work that lacked an understanding of the topic or requirements of the task
• learner work not at the required level (e.g. learner work that was too basic, descriptive or did not apply knowledge as required to meet learning outcomes)
• learner work with incorrect, irrelevant or superficial explanations, or inadequate research
• learner work that relied heavily on referenced materials, and demonstrated little processing by the learner
• irrelevant case studies used in assessments, especially when the New Zealand context was neglected
• poor English that inhibited learners from demonstrating their understanding or meeting the requirements of the New Zealand business industry context
• learner responses that were identical to other responses or the marking guide.

Poor assessment practice

Other issues of poor assessment practice noted by NZQA moderators include:

• assessment tasks not allowing learners to meet approved learning outcomes
• learners passing an assessment and/or module despite not meeting the minimum passing criteria for each learning outcome
• marking guidance that was not detailed or clear enough to allow for consistent marking
• assessors not following marking guidance
• over-generous marking
• assessors providing learners with little constructive feedback
• potential issues with authenticity, (e.g. heavy reliance on published material or copying another learner’s work)
• assessors not detecting learners’ use of synonym replacement software, which can make learners’ work meaningless.

Good assessment practice

Despite these many issues identified in this area, NZQA found some good assessment practices across the sector.

One good assessment practice instituted by a TEO was revising assessments before the next delivery to prevent incoming learners from accessing previous learner work. Another TEO mentioned that they were introducing alternative assessment sets for each course for the same reason.

Another positive move identified was that a TEO had implemented many actions to rectify issues identified in the 2017 NZQA moderation results. This TEO delivered both revised programmes, and the agreement rate for Level 5 was the fifth highest, and for Level 6 the highest agreement rate among all programmes monitored.

Many providers use case studies for students to apply their theoretical knowledge in simulated, real-world scenarios. One of the better examples of this was a TEO providing a clear, detailed, New Zealand case study that was relevant to the learning outcomes assessed. For example, a Human Resources course introduced a case study with a clear organisational chart, providing each staff member’s role expectations, along with detailed background. Because the provider used a New Zealand organisation, students were able to apply the relevant legislation. The NZQA moderator commented that the learners are provided with a realistic opportunity for how to implement New Zealand legislation in a human resource role.

In addition to being relevant to the New Zealand context, the best assessment materials:

• provided learners with the opportunity to meet the learning outcomes
• included a range of interesting and challenging assessment activities
• progressively allowed learners to build on the previous learning outcome and/or assessment
• resulted in articulate, thoughtful assignments or work-based application of skills that reflected the graduate profile.

As noted earlier, grades higher than a ‘C’ or ’50 per cent’ should mean that learners must achieve each learning outcome assessed. Some providers had this requirement clearly in their course outlines or programme documents. This was useful because it avoids any misunderstandings over marking for both assessors and learners.

Programme structure and delivery

This section of the monitoring report addresses:

• learning outcomes
• delivery methods
• programme length
• learning hours.

Learning outcomes and delivery methods

There were cases found where TEOs had changed approved learning outcomes without notifying NZQA. While changes to programmes are a natural outcome of programme review (as over time learning outcomes may be refined, combined or not needed), NZQA must be notified of changes. This ensures the programme still allows the learner to meet the graduate profile. Three programmes monitored contained unapproved changes to the learning outcomes.

Monitoring did not find any significant issues in delivery methods.

Programme length and learning hours

The main issue identified was the lack of formal monitoring or structure for learners’ self-directed learning hours. Monitoring found that TEOs relied heavily on the quality of learner assessment work to measure how they had spent self-directed learning hours. Therefore, it was unclear how TEOs ensured learners were meeting the approved learning hours.

There were two TEOs that used a study-log, which required learners to record what they studied outside of class. This is one of the effective ways to monitor learners’ self-directed learning. Another good practice used by a TEO was posting activities or discussion topics on a learning management system (e.g. Moodle), aside from assessments, to encourage learner research or reading. Tutors also monitored learners’ engagement through the learning management system.

While it is important to have breaks in study sessions, TEOs should avoid long or excessively frequent breaks. Those breaks could mean the TEO is under-delivering the programmes, especially when there was no evidence of how the TEO is addressing potential gaps between the self-directed and directed learning hours.

One TEO delivered programmes in a compressed format, which NZQA discourages.

Other unapproved changes to programmes

Five Level 5 and four Level 6 business programmes included unapproved changes, such as:

• adding a new course
• changing the length of courses
• changing assessment weightings.

TEOs must notify NZQA of any changes to an approved programme to ensure that it continues to meet approval and accreditation criteria.

Resources

This section of the monitoring report addresses:

• academic staffing and professional development
• teaching or learning facilities
• educational resources
• learner support.

Academic staffing and professional development

Monitors found that most academic staff were qualified to teach in the relevant subject areas. However, it was not always clear how academic staff maintained their expertise in and current knowledge of the subject.

While staff had research and further study activities in their professional development, New Zealand industry-specific engagement was often absent (e.g. being a member of a relevant professional organisation or attending a business-specific forum).

Monitors also found that many academic staff did not have, nor were they studying towards, an adult teaching qualification. Although this is not outlined explicitly in the rules, it is important that academic staff members should be properly trained in education or adult education, especially when NZQA identified problems in assessment and moderation. NZQA considers this type of professional development would help TEOs improve in that area.

Many TEOs conduct staff forums, workshops, or staff symposia to discuss teaching issues. These issues could include moderation practice, fair marking, and plagiarism detection. This is a positive example of professional development found during the monitoring period.

Providing learner support

Resourcing of learner support was consistently good across all programmes NZQA monitored.

TEOs supported learners before they started the programme, offered an orientation programme for a few days or week at the start of the programme and provided ongoing pastoral support for learners.

Some TEOs also offered academic writing, English language support and career guidance to learners outside of the programme requirements.

Providing appropriate facilities and resources

Teaching and learning facilities and education resources were also appropriate for the programmes NZQA monitored.

NZQA notes that TEOs are increasingly using online learning management systems (LMS), such as Moodle, to facilitate learning in and out of class.

An LMS can be used to:

• share learning resources
• facilitate learning activities (such as quizzes and discussion forums)
• track learner participation and engagement.