Research & Publications
Evidence-based AI methodology for vocational education
ConCOVE Tūhura Project
As Technical Lead, I ran New Zealand's largest research study on AI-generated assessments for vocational education.
The question was straightforward: can AI generate assessments that meet national moderation standards? The answer turned out to be more interesting than a simple yes or no.
AI-Generated Assessments for Vocational Education and Training
114-page peer-reviewed research report. Documents the methodology, findings, and ethical framework.
The Finding That Matters
Baseline AI assessments failed expert moderation. That wasn't surprising — generic AI outputs rarely meet specialised quality standards.
But when we applied the synthetic persona methodology to personalise assessments for specific learner needs, something different happened. Every single personalised variant achieved expert approval.
Independent moderators called the results 'excellent', 'appropriate', and 'way beyond minimum viable product'. The personalisation wasn't just acceptable — it was better than what we started with.
The implication: AI can meet compliance requirements for vocational assessment, but only with the right methodology. Generic approaches fail. Structured, human-directed approaches work.
Five-Phase Methodology
A systematic approach to AI assessment development that consistently passes expert review
Selection
Identifying appropriate unit standards and assessment contexts. Not everything is suited to AI generation — this phase defines where AI adds value.
Generation
Creating baseline assessments using structured AI prompts and quality controls. The prompts matter as much as the AI model.
Expert Review
Independent moderation by subject matter experts against national standards. This is where baseline assessments typically fail.
Personalisation
Applying synthetic personas to adapt assessments for diverse learner needs — ESL, neurodivergent, culturally specific contexts. No real learner data used.
Validation
Second expert review confirming personalised versions meet standards. This is where the methodology proves itself.
Synthetic Persona Methodology
The core innovation. Synthetic personas are representative fictional learners that enable AI to personalise assessments without exposing real student data.
Here's the problem we solved: AI personalisation typically requires feeding learner data into AI systems. That creates privacy risks, violates data sovereignty principles, and erodes trust. Training providers want personalisation benefits but can't afford those risks.
Synthetic personas break that trade-off. The AI works with fictional learners that represent real needs — ESL learners, neurodivergent learners, learners from specific cultural contexts. The personalisation is genuine. The privacy risk is zero.
We also discovered something unexpected: AI excels at generating assessor guidance tailored to specific learner needs. For neurodivergent learners, for example, the AI produced guidance like 'present one task at a time with clear beginning and end points' and 'allow 30-50% more processing time for verbal instructions'. Assessors can now support diverse learners effectively, even without specialist training.
Ethical Framework
Developed in partnership with George Angus Consulting, the ethical framework addresses the specific concerns of AI in vocational assessment.
The framework aligns with Te Tiriti o Waitangi principles and indigenous data sovereignty requirements. It was developed through consultation with Māori education stakeholders and builds in protections for cultural safety and data governance.
Key elements: transparency about AI use, bias mitigation at each stage, clear human accountability, data sovereignty compliance, and governance structures that keep institutions in control.
The framework is documented in the full research report and has been presented at research symposiums and industry events.
Publications
AI-Generated Assessments for Vocational Education and Training
The full peer-reviewed research report documenting methodology, findings, and ethical framework.
Download ReportAI-Generated Assessments in TVET: Achieving NZQA Compliance Through Personalisation
Conference presentation on the research findings and methodology.
Ethical Frameworks for AI in TVET
Framework development documentation and implementation guidance.
Speaking & Presentations
I present on AI in vocational education at conferences and industry events:
NZVETRF 2025 — AI-Generated Assessments in TVET: Achieving NZQA Compliance Through Personalisation
NZATD Webinar Series — AI in eLearning Design and Development
ConCOVE Research Symposium — Synthetic Personas and Data Sovereignty
For speaking enquiries, get in touch.
Research Collaboration
Interested in the research? Exploring AI for your own assessment context? Planning a study? I'd welcome the conversation.