How mistake-aware tutoring improves math learning for dyscalculia

NumPath, an AI-powered math tutor designed for children with dyscalculia, has taken a significant step toward more adaptive learning. Its core engine uses Bayesian Knowledge Tracing to estimate a student’s mastery of specific skills and selects the next problem accordingly. However, a recent update transformed the system from a generic difficulty slider into a true intelligent tutor—by making every mistake count.

From logging mistakes to acting on them

Early in its development, NumPath’s MistakeClassifier accurately tagged every incorrect answer with structured codes like BORROW_SKIP, DIGIT_REVERSAL, or MAGNITUDE_MISJUDGE. These labels were reliably stored in the database, but the adaptive selection engine ignored them entirely. The GetNextProblemUseCase component continued to rely solely on the student’s mastery estimate (p_mastery) to determine the next problem, regardless of recurring error patterns.

This approach conflicted with educational research principles, particularly the “Error as Diagnostic Signal” concept proposed by MacLellan and colleagues. It suggested that repeated mistakes should directly influence the type of problems presented next—something NumPath initially failed to do. A student who repeatedly made the BORROW_SKIP error could still receive the same skill-level problems after each attempt, with no targeted remediation.

A rule-based fix with explainable outcomes

To close this feedback loop, the team introduced a lightweight rule engine that monitors recent mistake patterns and triggers responsive actions. The design centered on three key parameters, each expressed as named constants for clarity and future calibration:

MISTAKE_WINDOW = 3          # Number of recent mistakes to analyze
THRESHOLD = ceil(MISTAKE_WINDOW / 2)  # Minimum occurrences to confirm a pattern
MISTAKE_KC_MAP = {
    "DIGIT_REVERSAL": "PLACE_VALUE",
    "BORROW_SKIP": "SUB_BORROW",
    "MAGNITUDE_MISJUDGE": "PLACE_VALUE",
    "PLACE_VALUE_CONFUSION": "PLACE_VALUE",
    "OPERATION_CONFUSION": "OPERATION_SIGN",
}

When a pattern is detected—such as two BORROW_SKIP errors within the last three attempts—two things happen immediately:

• Skill override: The engine prioritizes the linked knowledge component (KC), even if other skills have higher mastery scores.
• Difficulty adjustment: The problem difficulty drops by one level (0.2), but never below the entry threshold (0.3), preventing over-scaffolding.

The system also generates a clear explanation for every decision. The NextProblemResponse now includes a reason field like:

"Remediation: BORROW_SKIP detected 2× on SUB_BORROW (p_mastery=0.41)"

This transparency allows teachers to review a student’s session history and understand why a specific problem was chosen, aligning with the “Teacher-in-the-Loop” principle from the same research.

Why this change matters for learning outcomes

Before this update, NumPath’s adaptive engine operated more like a worksheet generator—repeating problems when a student struggled, without distinguishing between different types of errors. While difficulty adjustment based on streaks provided some personalization, it lacked meaning when the root cause of failure wasn’t addressed.

Now, every MistakeEvent record contributes to the student’s next experience in a traceable chain. A repeated DIGIT_REVERSAL no longer just logs a failure—it redirects the student to targeted place-value practice. This shift transforms the system from a reactive difficulty adjuster into a proactive tutor that learns from each mistake.

The change also strengthens the foundation for NumPath’s upcoming randomized controlled trial (RCT). The central hypothesis—that adaptive, mistake-aware tutoring improves outcomes for dyscalculic learners—can now be tested against a baseline of static worksheet-style practice. Without this mechanism, the comparison would be less meaningful.

Lessons learned and next steps

The implementation itself was minimal—just 60 lines of code across two files—but revealed important design lessons. One oversight was the absence of an index on the MistakeEvent table. Without it, querying recent mistakes for a student became inefficient. A simple composite index on (student_id, created_at) solved the issue, a reminder to prioritize query patterns early in development.

Another challenge was setting the detection threshold. Initially, the team considered requiring three-of-three matching mistakes to confirm a pattern. However, this proved too rigid. A student alternating between BORROW_SKIP and DIGIT_REVERSAL would never trigger remediation, despite a clear signal. The two-of-three rule catches patterns earlier, at the cost of occasional false positives. Without real student data, this remains a hypothesis—one the team plans to validate in Phase 4 of the project.

Looking ahead, the team is integrating knowledge component states into the teacher dashboard. Educators will soon be able to view not just weekly accuracy, but mastery estimates per skill. This final piece completes the feedback loop, ensuring that both the system and the teacher can make informed decisions together.

As NumPath continues to evolve, the core principle remains clear: logging mistakes is only the first step. The real value comes from using those mistakes to shape the next learning experience—and making every decision explainable.