Digital external assessment (formerly referred to as NCEA online) delivers digital exams to complement the long-standing paper-based approach. We are now looking to identify and take up the new assessment opportunities offered by digitally enabled teaching and learning.
This work will help us consider how we might deliver future digital exams. 21st century assessment can contribute to the equity of NCEA outcomes by engaging students in new ways, stimulating changes in teaching and learning and providing new ways for students to demonstrate their knowledge.
NZQA’s vision (depicted below) for equitable, innovative, credible and robust digital assessment, involves us working with education leaders, teachers, students and education agencies to co-design the approach.
Read more about NZQA's digital assessment vision
The following innovation and research work may result in further research, trials, pilots or enhancements to digital external assessment and the way we manage it over the next few years. We will update this section as we progress this work.
On this page
Innovation trials in assessment practice
Te Reo Rangatira
What is involved: During 2020, selected schools and kura which offer Te Reo Rangatira trialled a modified digital assessment which let students listen to instructions, questions and resources, as well as reading them.
Why we are doing this work: Māori students have told us they would like to hear as well as read the content of an exam. We are using Te Reo Rangatira to first test this idea in the context of mātauranga Māori. This may enable Māori students to have a richer and equitable digital exam experience.
Progress update: Following completion of the innovation trial, all levels of the Te Reo Rangatira 2021 digital practice exams included a new voiceover option which let ākonga listen to, as well as read, the exam instructions, questions and resources. The Level 2 practice exam also included an audio file for the listening standard. There are plans to introduce the new audio functionality for all Te Reo Rangatira end of year exams in 2022. More information is available in the summary report (PDF, 359KB) (external link).
Digital Statistics and Maths (Calculus) Assessment
What is involved: For subjects such as Statistics and Calculus which are hard to assess digitally we are trialling third-party software used by some schools. The trial involves the current assessment platform and assessment methods.
Why we are doing this: We want to provide digital assessment across all subjects and provide a good user experience for subjects difficult to do digitally. If students can use familiar software (that they have been using for their learning), it could increase their engagement and attract a broader range of candidates.
Progress update: We have completed and analysed the two maths innovation trials carried out in 2020. More information is available in the Digital Mathematics report (PDF, 851KB) (external link) and Digital Statistics report (PDF, 912KB) (external link). (external link)
Squirm Germ Gamified Assessment
What is involved: Biology Achievement Standard 90927 has been developed into an assessment based on a game approach. In 2020 selected schools will participate in a trial with students attempting the assessment and recording their responses.
Why we are doing this work: We want to find out whether assessments based on a game design have more applicability in the future, for example to enable candidates to better show what they know, potentially through being engaging and authentic.
Progress update: The innovation trial of the biology assessment based on a game design met the requirements of the standard, and there was a high level of interest and engagement from students and teachers. More information is available in the summary report (PDF, 287KB) (external link).
Research into assessment practice
Te Reo Māori (TRM) text-to-speech and spellcheck
What is involved: In 2020 we worked with Waikato University to investigate offering spellcheck and text-to-speech in te reo Māori. Their researchers worked with others to review high quality digital kupu (word) lists to support these functions.
Why we are doing this work: Providing spellcheck and text-to-speech functionality in te reo Māori can improve equity of access and outcomes for te reo Māori speakers, better reflecting how they learn.
Progress update: Waikato University’s research for NZQA is available here in the text-to-speech report (PDF, 2.1MB) (external link) and the spellcheck report (PDF, 494KB) (external link).
Text-to-speech
What is involved: NZQA has completed two trials with students to test an inbuilt text-to-speech function on the NCEA Online platform called ‘Polly’. In 2020 students were asked to test Polly and complete a survey. In 2021 we gained further insights from the students’ testing of the text to speech function, and from our discussions with students, SENCOs and teachers. Students’ feedback about how we could improve Polly included changing the voice, an option to control the speed that text is read aloud at and being able to turn text-to-speech on and off.
Why we are doing this work: Students who sit NCEA Online do not have the option to listen to assessment text being read aloud in digital assessments. Using this technology will improve accessibility for all students (including those who have difficulty with reading, have learning differences such as dyslexia, low vision or prefer aural learning) and give them greater control over how they complete their assessments.
Progress update: In 2022 we’re planning a further trial of Polly with students participating in the NCEA Literacy and Numeracy digital pilot assessments, scheduled for Terms 2 and 3. This will help us understand students’ experience of using Polly in live assessment conditions before making it more widely available for all eligible digital assessments. We’ll use the results to further refine improvements that can be made to Polly (expected to happen later this year).
Read more about the 2021 Text-to-Speech (Polly) evaluation report (PDF, 521KB) (external link).
Special Assessment Conditions - assistive technology
What is involved: We will further investigate technology being used in New Zealand to support students with Special Assessment Conditions (SAC) so we can plan for innovation trial(s) to test the compatibility of widely used applications with the exam platform.
Why we are doing this work: Allowing students in digital exams to use the assistive technology they already use to support their learning will increase equitable access and outcomes.
Progress update: In 2021 we worked with schools and students to understand their experiences with using assistive technology, SAC and exams. However, progress was affected by COVID-19. We’ll report on student experience of using this technology in early 2022.
NZQA is considering how to use enhanced accessibility of the online platform for assistive technology during second quarter 2022.
Hidemore informationSpecial Assessment Conditions - software stock take
What is involved: This research will identify technology being used in New Zealand to support students with Special Assessment Conditions (SAC) so we can plan for innovation trial(s) to test the compatibility of widely used applications with the exam platform.
Why we are doing this work: Allowing students in digital exams to use the assistive technology they already use to support their learning will increase equitable access and outcomes.
Progress update: We have completed a stock take of assistive technology tools (which includes application, purpose, suppliers, platform), who is using them, and what further research and analysis is needed for NZQA to consider approving use of assistive technology in digital assessments. More information is available in the summary report (PDF, 462KB) (external link).
More than one assessment opportunity a year
What is involved: We are researching how we can use the exam platform so NZQA and schools could offer digital exams at various times during the year.
Why we are doing this work: There is strong demand from schools to offer digital assessments, and there is potential for NZQA to stage exams at different times of the year so students can be assessed closer to the time they undertake their learning.
Progress update: In 2020 NZQA worked with the University of Waikato to complete a general literature review and synthesis of key findings from research studies. More information is available in the summary (PDF, 346KB) (external link)and full report (PDF, 1MB) (external link).
Remote supervision tools
What is involved: This research aimed to identify remote supervision tools suitable for NCEA digital exams and assessments to see what opportunities these could offer for students and NZQA.
During 2020 we developed a list of the available tools, their capabilities and how they are used in assessment settings.
Why we are doing this work: This could lead to greater flexibility and resilience of exam and assessment delivery, increasing the types of places and geographical locations where students can be digitally assessed, and supporting human supervision with new tools.
Progress update: In 2021 we completed a product review as a first phase of the technical Proof of Concept. We decided not to progress with any trials following the product review. What we learnt from the review will be used to refine requirements during 2022.
Automated marking
What is involved: We have been exploring how well-suited automated marking of essay questions is to the NCEA context; what is involved in set up and how marking software works in practice.
During 2020 the University of Alberta – Centre for Research, Applied Measurement and Evaluation (CRAME) used some past NCEA digital essay responses to complete automated marking, with the results reviewed and analysed by NCEA markers. Read about the work done in 2020 in the summary (PDF, 330KB) (external link)and full report (PDF, 1.5MB) (external link).
While the 2020 trial helped NZQA understand the practical implications of implementing an AES, we wanted to know more. In 2021 we completed a second AES trial with additional vendors.
Why we are doing this work: There was uncertainty about whether some of the issues identified in the 2020 trial were related to CRAME’s AES system, or to AES technology in general. We have trialled automated marking in 2021 with additional vendors so we could identify the cause of the issues, further broaden NZQA’s knowledge on AES technologies and consider how they could be applied in the NCEA context.
Progress update: More information about the 2021 trial is available in the summary (PDF, 164KB) (external link) and the two vendor reports, AES Trial 2021 – Final Report – Rembiont (PDF, 676KB) (external link) and AES Trial 2021 – Final Report – New Data Solutions (PDF, 2.3MB) (external link)
Scoping new services
Alternative delivery methods
What is involved: We are working with schools in New Zealand, Niue and the Cook Islands which lack access to reliable internet connections, a barrier for participating in digital exams. Once we understand the environment, we will trial alternative methods of delivering a digital exam that best fits the schools’ needs and integrates with the digital exams. The trials will be run outside the end of year exam period.
Why we are doing this work: This could enable schools without access to reliable internet connections to participate in digital exams and may also add resilience into the system by supporting the delivery of digital exams in the event of major incidents or natural disasters.
Progress update: This work was paused in 2020 due to COVID-19. In 2021 Niue participated in digital practice exams for the first time, and plan to participate in NCEA Online 2021 end of year exams. Niue is now out of scope for this project. Testing with the Cook Islands is underway.
Externally sourced assessment – digital practice exams delivered on NZQA’s online exam platform
What is involved: We are exploring how NZQA’s online exam platform could be used in schools so teachers can develop and deliver practice exams and activities.
Why we are doing this work: If the online exam platform could be used in schools, it would help teachers and students become more familiar with digital assessment and the platform.
Progress update:
In 2021 NZQA worked with four subject associations so that schools could offer digital practice exams for English, Agriculture and Horticulture Science, Classical Studies and Media Studies at all levels using NZQA’s online exam platform. Digital practice exams in Te Reo Māori and Te Reo Rangatira were also offered at all levels.
We evaluated feedback from the formal surveys sent to schools and students about their 2021 experience of participating in digital practice exams. More information is available in the summary report (PDF, 525KB) (external link) and summary dashboard (PDF, 587KB) (external link).
Planning is underway to provide externally sourced digital practice exams from 15 August to 30 September 2022.
NCEA submissions
What is involved: This is looking at using NZQA’s online platform to mark externally assessed work not done by examination. We will research what solution is needed for submitting work for NCEA external standards where assessment is not administered as an exam.
Why we are doing this work: Students are starting to generate portfolios digitally. We want to provide a way for teachers to upload school-based content and submit it for assessment by NZQA markers, removing the need for printed materials.
Progress update: We're designing and implementing a new end-to-end submission process as part of the Customer Experience Improvement Programme.
Scholarship online
What is involved: During 2020 we carried out preliminary research to establish the case for a limited trial of a digital Scholarship exam.
Why we are doing this work: Increasing numbers of students have completed their NCEA external exams and course work digitally and may expect to complete their Scholarship exam in the same way. We want to extend the digital external assessment opportunities open to senior secondary students to include Scholarship, as well as NCEA exams.
Progress update: We are trialling Scholarship Media Studies as a digital end-of-year exam in 2021. More information is available in the 2020 research summary report (PDF, 145KB) (external link).
Digitally-delivered Common Assessment Tasks
What is involved:
We have been investigating the feasibility of offering one or more of the Digital Technology Common Assessment Tasks (DCAT) on the NCEA Online platform in future years.
Why we are doing this work: Increasing numbers of students have completed their NCEA external exams digitally and may expect to complete other kinds of assessment in the same way.
Progress update: Our investigation is continuing, and we expect Digital Common Assessment Tasks (DCAT) will be offered on the NCEA Online platform in 2022.
Te Waharoa Ākonga | Student Services
What is involved: We will redesign and implement the student portal to improve students’ experience.
Why we are doing this work: This will give students direct access to personalised information about the digital exams they are entered in, reduce the time it takes to log in and access the correct exam and reduce the need for any assistance from supervisors and ECMs at the beginning of a digital exam session.
Progress update: We’re analysing customer feedback as we continue working on the design of the portal. This work is part of the Customer Experience Improvement Programme.
Designing new functionality
Digital tools and applications
What is involved: There are subjects that are ‘hard to digitise’ such as maths, chemistry, dance and drama, which need to be managed and delivered in a different way. We are identifying suppliers who can provide digital tools and applications to ensure an equivalent assessment experience to NCEA external paper exams. The digital tools and applications need to support a range of tasks such as: writing complex formulae and equations; creating and modifying tables; drawing maps; and identifying special requirements for music.
Why we are doing this work: We want to be able to offer a full suite of digital exams (at least at the substitution level) where possible in 2022. It’s important that using any digital tool is easy and instinctive for students so that they can focus their efforts on demonstrating their knowledge of the subject, not of the tool. Any future trials will include user experience testing to ensure the tools meet the needs of students.
Progress update: In 2021 we commissioned research from Massey University to investigate the use of e-ink (digital pen) in assessment. The findings report is due mid-2022.
We’re working on a proposal for a spread-sheeting functionality trial which is expected to take place by mid-2022.
Scanned paper responses
What is involved: During 2020 we completed a trial that scanned a selection of completed past exam paper booklets and processed them through the digital marking platform. This allowed marking to be completed digitally.
Why we are doing this work: We are aiming to bring together paper and digital marking and reduce paper interactions within the exam cycle. If this trial leads to scanning all paper responses, they will be marked and returned to students using the digital systems. It would eliminate the current two-week lag to return paper responses to students after the release of results and reduce the need to have separate quality assurance mechanisms for paper and digital responses.
Progress update: The 2020 trial gave NZQA confidence that scanned responses can be marked in the digital marking application RM Assessor3 and that the process is fit for purpose. Recommendations from the trial include preparing a brief for the next phase of this work and further scoping the requirements. This will be done along with benchmarking and check marking during 2021. More information is available in the summary report (PDF, 137KB) (external link).
Enhancing benchmarking and check-marking
What is involved: NCEA digital exams are marked online using the RM Assessor3 e-Marking tool. We used 2019 digital responses to test the benchmarking and check marking feature in the marking application of the digital exam platform. This enables all marking processes (for both digital and scanned paper responses) to be performed in the one digital environment. The trial may result in piloting the wider use of the digital marking tool.
Why we are doing this work: Any resulting improvements to marking information accuracy will help sustain confidence in the assessment system. Benefits would come from more streamlined delivery of candidate responses to markers and new ways of doing quality assurance.
Progress update: In 2020 panel leaders and check markers completed their trial benchmarking activities using RM Assessor3, finding that overall the functionality trialled is useful to carry out benchmarking and quality assurance processes digitally. Recommendations from the trial include progressing digital benchmarking and check marking along with scanning during 2021. More information is available in the summary report (PDF, 110KB) (external link).
Marker responses to students’ digital and scanned exam responses
What is involved: We completed a literature review in 2020 to investigate marker preference for, and perception of, different modes (i.e. marking scanned responses onscreen, marking paper-based responses, and marking digital responses). We have now done further research involving markers using NZQA’s marking system to mark scanned handwritten and digital responses onscreen.
Why we are doing this work: In 2021, we will scan NCEA Level 1-3 end-of-year paper exam responses so they can be marked on screen. To support the introduction of this process, a research project was proposed to understand how markers respond to digital (typed) and scanned (handwritten) exam responses.
Progress update: During 2021 we invited a panel of markers with experience marking both paper and digital exam responses to participate in the research. We collected data from group and individual interviews before and after onscreen marking commenced, and from the outcome of marking.
By the end of the research, the participants’ perceptions about onscreen marking had shifted towards being strongly positive. Participants identified key benefits as removing administration associated with paper-based exam responses, improved efficiency, greater responsiveness when checkmarking, and the ability for markers to collaborate in real time.
More information is available in the 2021 research report (PDF, 747KB).
Hidemore information
Assessment Master planner
What is involved: This trial will assess some of the features in the digital assessment platform to support the planning and provision of digital exam centres. In addition, this trial will assess the reporting capability for digital exam management, attendance, notification and messaging features.
Why we are doing this work: Testing this new system will allow us to decide if it is ready for use in the digital NCEA exams. This will improve efficiency and the exam experience for Exam Centre Managers (ECMs) and students.
Progress update: The trial has shown that none of the tested basic functionality entirely met the needs of NZQA or Exam Centre staff. Next steps involve better understanding the needs of the groups who benefit from the platform features trialled; and an analysis to match the AM Planner functions and needs of potential user groups. More information is available in the summary report (PDF, 111KB) (external link).