Introduction
Download a PDF version of this report [PDF, 228 KB]
The New Zealand Certificate in English Language (General/Workplace/Academic) (Level 4) [Ref:1883-1] was first published in 2012.
Following a mandatory review and extensive stakeholder consultation, the contextualised qualifiers were removed, and an academic contextualisation was developed.
The resulting New Zealand Certificate in English Language (Academic) (NZCEL) (Level 4) [Ref:1883-2] was published on the New Zealand Qualifications Framework (NZQF) in 2017.
NZQA selected programmes leading to the award of the NZCEL (Level 4) qualification as a monitoring priority. This is because NZCEL is listed in the NZQF Programme Approval and Accreditation Rules 2018 (PAA Rules 2018) as one of the internationally recognised proficiency outcomes that can be used to demonstrate English proficiency for enrolment into higher-level programmes and many vocational and undergraduate programmes/courses and specialised fields of study.
Previous NZQA monitoring of higher-level programmes (e.g. Business Diplomas) had identified issues in the English Language proficiency of some learners.
Thirty-six providers hold accreditation to deliver NZCEL (Level 4) and there is ongoing programme monitoring for this qualification. International learner numbers increased steadily over 2018–2019.
The purpose of this monitoring summary report is to:
- outline the key findings of the monitoring activities
- provide guidance to ensure programmes leading to NZCEL (level 4) continue to meet the programme approval and accreditation rules.
Programme Approval, Recognition and Accreditation Rules
The monitoring summary report is primarily for:
- tutors and assessors
- programme developers and programme leaders
- academic and quality managers.
See the top tips for tertiary education organisations (TEOs).
On this page
Programme monitoring of the NZCEL (Level 4)
NZQA’s programme monitoring for the NZCEL (Level 4) is based on an evaluation of evidence to establish the extent to which the programme continues to meet the PAA Rules 2018.
Programme monitoring also provides assurance that TEOs are continuing to meet the criteria set out in the NZCEL academic qualifier and the English for Academic Purposes (EAP) unit standards, which are a mandatory component of the NZCEL Level 4 qualification.
Programme monitoring activities
NZQA used two types of programme monitoring for NZCEL (Level 4):
Full programme monitoring
This involved an onsite visit to the delivery site(s), facility tour, class observations, interviews with staff and learners, review of the QMS document and validation of assessor decisions (moderation).
Moderation only
This involved validation of assessor decisions and review of the programme document carried out remotely.
For the moderation aspect of NZCEL programme monitoring, NZQA used a ‘learner journey’ approach. NZQA randomly selects learners and moderated all their assessments that contributed to the award of the qualification. This provides a holistic view of the quality of assessed work for each learner, in relation to the approved learning outcomes.
From 2018 to 2020, NZQA monitored 32 NZCEL (Level 4) programmes delivered by 31 TEOs as below (including a second ‘moderation only’ activity for one TEO):
- 12 full programme monitoring
- 20 moderation only.
Overall outcomes of NZQA’s programme monitoring activities
Of the 32 NZCEL (Level 4) monitoring activities:
- seven programmes met the programme criteria overall
- a further nine programmes met some programme criteria
- the remaining 16 programmes did not meet the programme criteria overall.
NZQA identified a range of issues preventing TEOs from meeting all programme criteria.
The significance of the issues NZQA identified informs the appropriate follow-up action required by the TEO and NZQA.
In most cases where TEOs met some programme criteria or did not meet criteria overall, NZQA expected TEOs to implement the recommendations and any requirements from the programme monitoring and moderation reports.
NZQA also required additional remedial action from four of the organisations that did not meet programme criteria overall, including imposing statutory conditions on programme delivery in one instance.
Common issues
The most common issues which prevented programmes meeting programme monitoring criteria included:
- poor assessment and moderation practice
- assessment tasks do not meet the sufficiency requirement of the standards
- assessments did not make up the credit requirements of the module
- unapproved changes to the programme
- poor version control of the unit standards
- a lack of robust programme review.
Programme structure and delivery
NZQA’s programme monitoring found that all TEOs had appropriate delivery methods for the NZCEL (Level 4).
TEOs used a thematic-based approach which was delivered through both face-to-face and self-directed learning.
Student intakes ranged from two to seven, depending on the TEO.
Delivery methods
TEOs monitored have well-timetabled hours that include a range of delivery methods such as:
- lectures
- tutorials
- workshops
- group work
- library work
- use of online teaching resources via digital learning platforms
- debates
- field trips.
Programme length
The programme duration of TEOs monitored was between 14 to 20 weeks.
The programme length of most TEOs is as approved by NZQA, and the monitors were provided with evidence of learners’ attendance.
Programme structure
All TEO’s programmes consist of four components worth 15 credits each.
Some TEOs divided the programme into three blocks of delivery (six weeks in each block) or two 8-week blocks (three courses in each block).
There is often a one-year limit for completion of the qualification.
The TEOs monitored have delivered the programme as four skills, that were either integrated or blended courses of the skills (listening, speaking, reading and writing).
TEO programmes are composed of either all unit standards or a mix of unit standards and local components.
The main structural issue identified in the monitoring was TEOs with rolling intakes. This disadvantaged the learners in several ways.
Learners who started the programme during the delivery of productive skills like writing or speaking without completing receptive skills of listening and reading were disadvantaged on their assessments.
In an assessment-heavy structure, new learners are often required to complete multiple assessments within the first four weeks of delivery.
One TEO had rolling intakes every Monday which meant that sometimes new learners were placed in a classroom with other learners already at the end of the programme.
Learning outcomes
Some TEOs failed to inform the learners of the learning outcomes of the programme components or modules.
Learners were able to describe their progress within the course but could not explain it in terms of learning outcomes.
It is critical for learners to know what skills and knowledge they should be acquiring and how the learning outcomes in their assessments are related to language acquisition.
Two TEOs had discrepancies between the programme documents, student handbook and assessments regarding the learning outcomes. These unintentional errors could misinform the learners on what learning outcomes they are being taught and assessed against.
One other TEO had partially updated the versions of the unit standards leading to changes in the assessment weightings and credit without notifying NZQA. Notifying NZQA of these changes ensures the programme still allows the learner to meet the graduate profile.
Learning hours
Most of the TEOs’ specified total learning hours, teaching hours, teaching weeks, and learning activities were consistent with those approved.
However, for two TEOs, NZQA’s evaluation found discrepancies from the approved programme document where some face-to-face delivery was substituted with self-directed learning. Effective means of monitoring self-directed learning and highly structured expectations regarding what is to occur during this time is important for learner motivation and progress.
NZQA’s monitoring also identified three instances of TEOs significantly underdelivering the approved 600 learning hours, which could impact the quality of the programme delivery, consequently affecting learner performance.
Programme regulations
This section of the programme monitoring report focuses on how TEOs applied the regulations for admissions and normal progression within the programme.
Admission
All TEOs have clear, relevant, and appropriate regulations for admissions.
However, three TEOs were not able to apply the appropriate programme regulations requirements for admission for their international students.
NZQA considers appropriate regulations for admission include:
- ensuring learners’ placement test scores meet the requirements of the programme
- admission requirements are clearly stated and adhered to, and there is a clear record for any discretionary admission decisions TEOs make
- effective processes are in place for checking the authenticity/validity of online IELTS results and other internationally recognised English Proficiency Outcomes for International students (as per PAA Rules).
Normal progression within the programme
All TEOs clearly indicated compulsory module-specific courses and the pre-requisites or co-requisite for normal progression within the programme.
However, for some TEOs, the delivery progression was variable depending on the campus delivering the programme.
NZQA expects that requirements for progression should be consistent across all campuses delivering the same programme.
Assessment and moderation
Moderation systems and processes
All TEOs had policies and processes for moderation, but eight TEOs did not fully implement or apply these policies and processes effectively.
Two TEOs did not have appropriate regulations that specify requirements for assessment methodology and procedures regarding how learners achieve the qualification and the authenticity of learner work.
NZQA’s programme monitoring findings showed the best moderation systems ensure:
- teaching staff/assessors receive professional development on assessment and moderation
- assessment tools allow learners to meet all learning outcomes
- feedback is gathered on the clarity of instructions and the appropriate level of assessment tasks
- assessment guides, marking schedules and answer keys provide clear guidance
- moderation activities are clearly scheduled in a regular cycle
- action plans arising from moderation reports are sufficiently detailed, implemented and monitored.
Issues found in this area included:
- tutors/assessors not involved in the design, development and/or moderation of assessments
- limited support for staff on current assessment and moderation practices
- serious weaknesses in the current assessment methodology
- limited and sometimes no evidence of moderation activities
- lack of or limited moderation activities between campuses
- unplanned resits and resubmission which removes or diminishes the learning opportunities for learners in a tight delivery timeframe
- excessive numbers of resits and resubmissions
- same tasks used in resits
- ineffective moderation documentation: issues, feedback or actions not clearly or appropriately indicated
- external moderators’ comments and/or recommendations not addressed.
Assessment methods
The assessment tools used by TEOs are:
- formative and summative assessments, including those for mandatory EAP unit standards and/or English Language unit standards
- naturally occurring evidence
- portfolios of assessment
- e-portfolios on online learning platforms.
NZQA moderation
Of the 32 programmes monitored, 21 did not meet the requirements of 6.1 Criterion 1 of the PAA Rules.
NZQA uses a ‘learner journey’ approach to select learner evidence for moderation: i.e. all assessments completed by the learner for the purposes of award of the qualification. Typically, NZQA selects four graduates, and moderates approximately 48 pieces of learner work in total.
NZQA moderates learner work at the programme module level (listening, speaking, reading and writing) and makes judgements based on whether the learner has provided sufficient evidence of meeting all four graduate profile outcomes and the approved learning outcomes for the qualification.
NZQA provides detailed feedback to TEOs on:
- assessor decisions
- assessment materials and design
- authenticity issues
- other issues that affect the credibility of the qualification awarded.
If serious concerns are raised in assessment practice, NZQA may consider statutory action to protect the integrity of qualifications awarded.
Assessor decisions
The agreement rate between NZQA moderators and the TEOs’ assessor decisions ranged from 8 per cent to 95 per cent.
The main reason for moderators not verifying assessor decisions were:
- overgenerous, incorrect and/or inconsistent marking or assessor decisions
- reformulation and/or self-correction permitted during assessments
- over scaffolding and excessive support is given to learners that contradicts the unit standard requirements and may not give learners opportunities to demonstrate their ability to meet the required standard independently
- rehearsed and scripted presentations and discussions limiting learner engagement with the audience
- learners provided with notes and summary notes before assessments
- learners awarded marks, although they did not meet the learning outcomes
- learner work not at the required Common European Framework of Reference (CEFR) level.
Assessment materials
NZQA expects that the learner and the tutor/assessor should have clear guidance on the level of performance required to meet each learning outcome.
Assessing learning outcomes that map to the graduate profile is central to assuring the quality and integrity of programmes.
Where the assessment task covers multiple learning outcomes, the assessor must ensure that the learner evidence meets the minimum passing criteria for each learning outcome.
The NZCEL Guiding Document was first published in September 2017 to provide information to support providers delivering the programme. In the previous version of the Guiding Document, the sufficiency requirements for the assessments were open to interpretation in terms of the minimum number of assessments for a programme leading to the award of the qualification.
In May 2019, the Guiding Document was revised to strengthen the guidance to TEOs delivering the programme.
NZCEL Guiding Document [PDF, 460 KB]
The main issues with assessment design and materials included:
- inconsistent and/or incorrect resit, resubmission or further assessment opportunities which do not meet the unit standard requirements or contradict the explanatory notes
- unclear and inconsistent benchmarks for passing assessments in marking schedules and answer keys
- assessment task and design provide limited or insufficient opportunities for learners to demonstrate competency independently
- inconsistencies between the learning outcome requirements and assessment tasks
- unclear assessment instructions such as allocation of time and preparation for assessment
- assessment tasks which do not allow learners to meet the sufficiency requirements of the graduate profile outcome
- assessment tasks which do not meet the level and academic qualifier of the qualification
- inconsistencies with the instructions and/or requirements of assessment tasks
- assessment design that requires different academic demands for similarly weighted assessments
- inconsistencies in the marking guidance and schedules
- assessment resources provided to learners are not authentic (do not match real-world context) or do not meet the level of requirement and/or sufficiency requirements of the unit standard
- discrepancies between the approved and the actual number of assessments
- discrepancies between the approved programme learning outcomes and the assessments’ learning outcomes
- misinterpretation of the unit standard requirements resulted in credit deficiency and insufficiency of assessments
- no evidence of credit values gained in each module
- incorrect assessment conditions.
Authenticity
All TEOs delivering NZCEL (Level 4) recognise that ensuring the authenticity of learner work is critical to the quality assurance process.
NZQA also notes that ensuring that learner work is authentic, is more than just identifying plagiarism at the time learners submitted their work. For example, one TEO used a listening assessment that was lifted from a published coursebook and used without modifying it. Learners may have had access to this material before the assessment.
Effective methods used by TEOs to ensure authenticity included:
- new assessments being used for each cohort, multiple versions of the same assessment and a different version of the assessment used for resit/reassessment opportunities
- workshops regarding paraphrasing, summarising and referencing to avoid learners cutting and pasting from sourced material
- use of plagiarism tools for formative and summative assessments. By using plagiarism tools for formative assessment, learners are able to become familiar with how it operates and identify how much of their work is copied
- ensuring learners are aware of the resit, resubmission or further assessment policies
- ensuring learners do not have access to electronic devices or mobile phones during assessment.
Good assessment practice
NZQA monitoring also found examples of good assessment practice across TEOs.
Good assessment practice included:
- the use of e-portfolios and/or formative assessments which show learners are able to meet the requirements independently
- individual learning plans which are discussed between each learner and the tutor on a weekly basis
- regular communication and collaboration between tutors at different delivery sites and regular cross-campus moderation
- team teaching where two tutors deliver different aspects of the programme
- assessment activities that provide learners with opportunities to meet all the required learning outcomes
- allowing learners who do not require reassessment to engage in a practical project instead of ‘wasting’ the last week of the programme
- a range of interesting and challenging assessment activities
- progressively allowing learners to build on the previous learning outcome and/or assessment
- enabling learners to individually meet the graduate profile outcomes of the programme.
Assessment support materials and resources for NZCEL (Level 4) EAP standards have been developed to support interpretation of the unit standards and to show how they can be meaningfully assessed.
NZCEL assessment support resources
There is further guidance for online assessments for NZCEL (Level 4), English for Academic Purposes and English Language unit standards on the NZQA website.
Resources
Of the 12 TEOs visited, 11 had the capability and capacity to support sustained delivery of the programme through appropriate academic staffing, teaching facilities, educational and physical resources, and support services.
Academic staffing
Monitors found that most academic staff employed by TEOs were highly qualified and experienced to teach the English language.
However, it was not always clear how academic staff maintain their expertise and the currency of their knowledge of the programme.
Staff interviewed by NZQA spoke highly of their employers’ investment in professional development.
Some employers provide in-house or external professional development programmes and/or training.
Other TEOs have provided study support to teaching staff to complete adult education qualifications.
Examples from well-resourced TEOs included:
- full-time, permanent academic staff members/assessors with specialist qualifications at a post-graduate level and who have New Zealand-based teaching experience
- staff teaching the programme have an adult education and/or assessment qualifications or unit standards and experience in competency-based assessment
- the TEO has appropriate academic staffing/learner ratios based on the programme delivery requirements and learner needs
- the programme is well supported by the Programme Leaders and senior management
- provision for leave or release from teaching duties to attend relevant external professional development training.
Teaching and learning facilities
In all 12 TEOs visited, teaching and learning facilities were appropriate for the programme.
NZQA notes that TEOs are increasingly using online learning management systems (LMS) such as Moodle, Canvas access, Vital English or Google as a platform/additional resource for learner activities.
NZQA also notes that all TEOs provide good teaching and learning facilities which include:
- computer laboratories with internet connections
- classrooms equipped with internet connections, a whiteboard, PC, sound system and projector
- an on-site library holding substantial physical and online resources, including electronic research databases. Some TEOs have a dedicated librarian for the NZCEL and other non-English speaking background programmes to provide additional support
- video room, listening laboratory and satellite TV available for learners.
Educational and physical resources
For all the TEOs visited, educational and physical resources were appropriate and adequate for the programme.
These resources include:
- communal areas and a café on site
- prayer room
- spaces designed for small to large groups which offer multiple break-out areas and different seating types for learners to work together, and also smaller bookable learning rooms.
However, NZQA found that some TEOs had limited and/or no resources in the classrooms.
In some cases, there were fewer than ten dictionaries available for learners, which meant they had to share dictionaries during assessments.
There was also a limited number of required textbooks that needed to be shared among two or more classes.
Learner support services
Learner support was appropriate and adequate for most of the TEOs visited.
Some of the good practices in this area included:
- orientation for each new cohort of learners, including a powhiri
- learners receiving support from all teaching staff, the Programme Leader and/or Director of Studies
- study skill advisor being available to all learners who needed extra support to improve their academic performance
- one-to-one consultations and regular workshops available for learners such as academic writing, study skills, referencing and exam preparation
- learners are referred to counselling services if required
- effective systems for identifying at-risk learners at an early stage and organising additional support, such as regular after-class meetings to help learners struggling to stay on track to complete the programme
- a 24-hour phone number available for learners to call in case of emergency
- student advisors who speak a variety of languages catering to the diversity of international learners
- information provided to learners of their rights under the Code of Practice and how to access internal and external grievance procedures
- a 24 hour, seven days a week IT hotline
- on-campus career advisors.
Programme review
NZQA’s programme monitoring found that this is the second weakest area next to assessment and moderation.
NZQA recommends that TEOs review their programmes at least annually, although interim review activities may feed into their annual review.
The annual programme review is an opportunity to evaluate and ensure whether the programmes meet the qualifications’ strategic purpose and outcome statements. Reviews must be thorough and robust to ensure that the approved programme is still fit for purpose and meets the need of stakeholders.
Of the 12 programmes monitored, nine did not meet the requirements of good programme review practices.
Ongoing reviews and monitoring the quality of outcomes
Ongoing reviews and self-review practice that were effective included:
- frequent discussions among tutors and academic leaders about learners as a result of feedback and surveys
- taking steps to assist learners at risk of dropping out or not achieving
- weekly tutor meetings throughout the course and final reflections, evaluation and planning before the next delivery
- reviewing the programme annually or after each cohort
- conducting several types of surveys to gather evaluative information (e.g. course surveys, first impression surveys and tutor surveys)
- producing annual reports on programme evaluation to address specific Key Evaluative Questions (KEQs)
- regular discussions on the theme of ‘what? so what? and now what?' that lead to outcomes like action plans with specific, time-bound improvements
- action on feedback from moderators, assessors and learners regarding assessments or assessment conditions.
Issues found in this area included:
- reviews not taking into consideration the impact of the assessment and re-sit/resubmission issues that have affected the balance of teaching, learning, and assessment in this programme
- TEOs not undertaking any formal programme review
- lack of clarity about how learner and/or staff feedback is acted upon
- no analysis of graduate destination data.
Programme update
Most TEOs update their programmes after each cohort based on learner interviews and evaluations of the programme. As a result of the reviews, TEOs often update and/or redevelop the learning outcomes, programme regulations and content of the programme document.
However, some TEOs did not maintain or keep the content up to date to align with the latest version of the qualification or unit standards.
Most TEOs have an adequate and effective review process to measure the programme’s performance as well as its capability to support the NZCEL (Level 4) programme.
Effective practice included:
- review of the programme leading to delivery changes and improvements for better programme management
- revision of the programme to meet the requirements of the NZCEL (Academic) Level 4 qualification (1883 version 2)
- early indications or concerns raised by learners and lecturers are addressed through programme redevelopment
- providing staff training in areas identified as concerns
- each annual programme self-assessment report includes a detailed and clearly set out action plan as well as a record of progress against the actions identified in the previous year’s report.
Top tips
- Provide professional development opportunities for academic staff and internal moderators to improve their skills in the areas of assessment design, assessment marking, and moderation.
- Implement internal moderation systems, with qualified staff quality assuring assessments materials and a proportion of assessor decisions.
- Ensure that the focus is on learning outcomes and NZQF level when designing NZCEL assessments, quality assuring assessment materials and marking learner work.
- Use NZCEL guidance documents and ensure unit standard requirements are met when developing and delivering courses and assessments.
- Assessment tasks must have clear statements to indicate what students are expected to know and able to do at the completion of a unit standard.
- Implement a system to oversee moderation processes that gives a clear picture of identified issues and how the TEO is addressing those issues.
- Conduct a robust programme review to critically evaluate whether the programme meets the strategic purpose and graduate profile of the qualification.
- Apply a broad range of prevention strategies to maintain academic integrity within the programme.
- Provide learners with further academic support outside of the notional learning hours to ensure graduates from the programmes meet the academic level needed for further studies and pathways.
- Students should be provided with opportunities to independently demonstrate competency against all the performance criteria stated in the unit standards.
- Clearly articulate the description of the level of attainment that acts as a stable reference point or recognised measure for assessor decisions on the quality of student work.
- Ensure that there are clear and consistent resit and resubmission policies that meet the requirements of the unit standards.