Why explainable AI beats raw accuracy in classroom teaching tools

Teachers using artificial intelligence tools often face a critical dilemma: high accuracy scores don't necessarily translate to effective learning support. When a dashboard displays "Emma — 43%" without context, educators can't determine whether the issue stems from place value confusion, borrowing mistakes, or foundational number sense gaps. The distinction remains hidden behind a single number, rendering the data practically useless for intervention.

From Opaque Stats to Actionable Insights

NumPath recently transformed their teacher dashboard to address this challenge by introducing a Knowledge Component (KC) mastery panel. Instead of presenting one opaque accuracy score per student, the system now displays color-coded progress bars for each skill. These bars expand to reveal mastery percentages, level labels (Novice/Developing/Mastered), and attempt counts. The key insight? Teachers need granular insights they can act upon, not just technically correct but ultimately meaningless numbers.

Designing for Real-World Classroom Needs

The team faced three primary approaches when redesigning the dashboard:

• Accuracy-only metrics: Fast to compute and easy to implement, but completely unactionable in practice
• Raw Bayesian Knowledge Tracing parameters: Provides complete data including p_mastery, p_learn, p_guess, and p_slip values, but overwhelming for classroom use
• KC mastery levels: Translates raw mastery percentages into intuitive three-tier labels with color coding while preserving detailed numbers for deeper analysis

The team ultimately chose the third option, balancing comprehensiveness with usability. Mastery thresholds were codified in get_kc_states.py as named constants:

_MASTERY_DEVELOPING = 0.40
_MASTERY_MASTERED = 0.80

def _mastery_level(p_mastery: float) -> str:
    if p_mastery >= _MASTERY_MASTERED:
        return "Mastered"
    if p_mastery >= _MASTERY_DEVELOPING:
        return "Developing"
    return "Novice"

A deliberate UX decision ensures students with no attempts appear as 0%/Novice rather than "no data." Teachers see the complete KC grid from day one, where empty bars indicate unencountered skills rather than data errors.

Secure, Role-Based Data Access

The system implements robust access control through a require_authenticated dependency combined with role-based logic in the route handler:

@router.get("/{student_id}/kc-states", response_model=KCStatesResponse)
async def get_kc_states(
    student_id: uuid.UUID,
    db: AsyncSession = Depends(get_db),
    auth: dict = Depends(require_authenticated),
) -> KCStatesResponse:
    role = auth.get("role")
    if role == "student" and auth.get("sub") != str(student_id):
        raise HTTPException(
            status_code=status.HTTP_403_FORBIDDEN,
            detail="Access denied"
        )

This centralized approach contrasts with scattered validation logic across multiple dependencies. Students access only their own KC states, while teachers can view any student's data, creating a clean separation of concerns.

Transforming Research into Teaching Practice

The MacLellan Intelligent Tutoring System (ITS) framework emphasizes the "Teacher-in-the-Loop" principle, which requires more than just providing data—it demands actionable insights. A simple 43% accuracy score might indicate struggle, but a KC panel revealing SUB_BORROW at 12% (Novice, 8 attempts) while PLACE_VALUE stands at 67% (Developing, 14 attempts) tells educators exactly where to focus intervention efforts.

This distinction transforms a reporting tool into an actual teaching instrument. The upcoming randomized controlled trial (Phase 4) will evaluate whether teachers using KC-level visibility intervene differently than those relying solely on accuracy scores. The dashboard serves as both research instrument and practical teaching aid.

Technical Lessons in Simplicity

The implementation team discovered that simplicity often trumps cleverness in educational software. While an ORM outer join might seem elegant, it introduces complexity in handling nullable columns. Instead, a straightforward approach using two separate queries plus dictionary lookup proved superior:

kc_by_skill_id = {record.skill_id: record for record in kc_records}
summaries = [
    KCStateSummary(
        skill_code=skill.code,
        p_mastery=round(kc_by_skill_id[skill.id].p_mastery, 3)
            if skill.id in kc_by_skill_id else 0.0,
        ...
    )
    for skill in all_skills
]

This strategy enabled nine unit tests to run in just 0.03 seconds without requiring a live database. The lesson? Keeping domain logic in dedicated use cases rather than inline route handlers creates more maintainable, testable code.

The Road Ahead: Deeper Student Insight

Phase 2 of the KC dashboard will integrate recent attempt history directly into the student detail panel. Teachers will see specific problems students answered incorrectly, complete with classified mistake codes. This enhancement will help educators identify emerging patterns as they develop, enabling even more targeted intervention strategies.

Core Principles for Educational AI Tools

The NumPath experience reveals three fundamental truths about building effective AI for education:

• Accuracy is output; mastery is signal – Single performance metrics fail to guide teaching decisions, while per-skill mastery states provide the minimum viable explainability for intelligent tutoring systems

• Empty data is informative, not broken – A 0% bar for a new student signals unencountered skills rather than missing data, eliminating the need for "no data yet" placeholders

• Two queries with a dictionary outperform complex joins – For small reference datasets like five core skills, simplicity in query structure and data transformation trumps sophisticated ORM operations

The future of classroom AI lies not in presenting increasingly accurate numbers, but in delivering insights teachers can immediately apply in their daily instruction.