What is human-over-the-loop AI?

Human-over-the-loop means humans design, direct, and control AI systems rather than just reviewing outputs. AI handles repetitive tasks while experts make decisions and maintain accountability. It's different from 'human-in-the-loop' where humans only check AI work after the fact.

How do you prevent AI hallucination?

Our methodology controls when AI can be creative and when it must follow strict structure. Human experts validate all outputs at defined checkpoints. This gives you AI creativity where it's useful, and tight control where accuracy matters.

How do you handle learner privacy with AI?

We developed a synthetic persona methodology that enables AI to personalise assessments without using real learner data. The AI works with representative fictional learners. The personalisation is real; the privacy risk is zero.

Hotloop.ai | Human-Over-The-Loop AI for Vocational Assessment

Q: How do you address AI bias in assessment?

We use structured validation methods at each stage of assessment generation. Independent experts review outputs against quality standards. Our methodology was tested in New Zealand's largest TVET AI study, where personalised assessments achieved universal expert approval.

What is Human-Over-The-Loop?

Most AI discussions focus on 'human in the loop' — the idea that people should review AI outputs before they're used. That's necessary. It's not sufficient.

Human-over-the-loop means something different. Humans design the systems, set the boundaries, define quality standards, and keep the authority to change course. AI handles the repetitive generation work. Qualified people make the judgments that matter.

In assessment contexts, this means assessors make final competency decisions. Experts define what quality looks like. Institutions retain control. The AI operates within boundaries set by humans, not the other way around.

The Questions Everyone Asks about AI

These are the questions I get asked most. Here's how the methodology handles them.

AI Bias

Structured validation at each stage. Independent expert review against quality standards. Our methodology was tested in NZ's largest TVET AI study — personalised assessments achieved universal expert approval.

AI Hallucination

Controlled creativity. The methodology defines when AI can be creative and when it must follow strict structure. Human checkpoints at each stage. You get AI flexibility where it's useful, tight control where accuracy matters.

Privacy & Data

Synthetic personas enable personalisation without exposing real learner data. The AI works with representative fictional learners. Personalisation is real. Privacy risk is zero. Built for data sovereignty compliance.

Accountability

Human-over-the-loop means experts direct and validate all outputs. Clear decision points throughout. You know who's responsible at each stage — and it's never just the AI.

Research-Backed

As Technical Lead for the ConCOVE Tūhura project, I led New Zealand's largest study on AI-generated assessments for vocational education.

The study tested whether AI-generated assessments, to meet national moderation standards and create a baseline for AI assessments, took that feedback and applied the synthetic persona methodology to personalise assessments for specific learner needs. Every single variant passed independent expert review.

Experts called the results 'excellent', 'appropriate', and 'way beyond minimum viable product'. The methodology is documented in a 114-page peer-reviewed report.

Download the Report More on the Research

Karl Hartley

AI Researcher & Vocational Education Specialist

20+ years in learning and development. I've worked across telecommunications, industry training, and educational technology. My focus now is AI-generated assessments — figuring out what works, what doesn't, and how to keep humans in meaningful control.

I ran the technical side of New Zealand's largest TVET AI research project. The methodology we developed there is what Hotloop.ai is built on.

Based in Kirikiriroa (Hamilton), New Zealand

Full Bio karlhartley.com LinkedIn

Research Partnership

George Angus Consulting

I work alongside Stuart Martin at George Angus Consulting on research and consulting projects. Stuart handles policy, quality assurance, and ethical frameworks. I handle the technical AI architecture.

We co-led the ConCOVE research and delivered the ethical framework for AI-generated assessments. For projects that need both technical depth and policy expertise, we work as a team.

georgeangusconsulting.com

Get in Touch

Whether you're exploring AI for assessment, planning a research project, or just want to talk through what's possible — I'd welcome the conversation.

karl@hotloop.ai

LinkedIn karlhartley.com