Can AI truly erase data? Building a memory audit tool to verify erasure

AI systems today are trained to remember—long context windows, persistent memory, and knowledge graphs ensure no detail slips through. But what happens when users demand deletion under privacy laws like GDPR or India’s DPDP Act? The answer is troubling: erasure rarely means true deletion.

That’s why I built Lethe, a polygraph for AI memory designed to expose hidden data leaks and verify real erasure. Instead of trusting that an AI forgot, Lethe actively probes its memory to confirm whether sensitive information remains accessible—even after a user requests deletion.

The erasure paradox: Why deletion isn’t enough

Privacy regulations such as GDPR Article 17 and India’s Digital Personal Data Protection Act (2023) grant users the right to erasure. Yet the AI industry often treats deletion as a one-time database operation, ignoring the ripple effects of derived data. When a user’s information is embedded into vectors, transformed into graph nodes, or cross-referenced across records, deleting the original row doesn’t remove these artifacts.

For example, if two customers discuss the same issue, erasing one customer’s data might still leave their identity exposed through the other’s records. The AI might claim to have forgotten, but the claim lacks proof. Lethe was built to demand that proof.

Turning recall into a weapon for verification

Lethe flips the script on AI memory testing. Instead of querying memory to retrieve information, it uses recall as an attack surface to expose leaks. A specialized Auditor agent fires a fixed battery of 15 extraction probes at the memory system before and after a deletion request. Each probe is designed to test different attack vectors:

• Direct probes: Ask for specific PII like phone numbers or addresses.
• Inference probes: Identify users indirectly by querying complaints or issues without naming them.
• Reconstruction probes: Pull full lists of sensitive data, such as all complaints exceeding a certain amount.
• Relational probes: Check whether deleted nodes still leak through graph edges, revealing hidden connections.

The probes are frozen—they remain unchanged between tests to ensure consistency. A judge LLM (powered by Anthropic’s API) scores each response as either LEAK or SAFE, with a deterministic fallback detector ensuring the process runs even without an API key.

The results: When erasure fails

The test began with a seeded customer profile named Ravi Sharma. Before deletion, all 15 probes leaked sensitive data, including phone numbers, emails, complaint details, and even social connections. After triggering a standard forget operation, the results were eye-opening.

• Record deletion alone (removing the raw dataset row) left residual references. Another customer’s transcript mentioned Ravi Sharma by name, embedding his identity in a different record. This artifact remained untouched, proving the erasure was incomplete.
• Full person erasure required a cascade deletion—removing not just the raw data but also redacting every cross-reference across the knowledge graph. Only then did the probes return 0 leaks, confirming true erasure.

Watching the scoreboard shift from red to green—15 leaks down to zero—was a rare moment of clarity in an industry often content to assume deletions work.

A tamper-proof certificate: Proof you can’t fake

A green checkmark on a screen isn’t enough for regulators or auditors. Lethe generates a Deletion Certificate that defends itself against tampering:

• Evidence-bound integrity: Every probe, its before-and-after responses, and the judge’s verdicts are hashed into a SHA-256 Merkle tree. The root hash is embedded in the certificate. Altering even a single probe result breaks the chain.
• Cryptographic signing: The certificate body is signed with HMAC SHA-256 over a canonical JSON form. Any alteration invalidates the signature.
• Independent verification: A verify endpoint recomputes both the Merkle root and signature. The UI includes a "Simulate tampering" button—changing a leak verdict to zero instantly reveals a broken signature.

This design ensures that the certificate isn’t just a digital receipt; it’s irrefutable evidence that the AI truly erased the data.

From prototype to real-world testing: The Cognee Cloud insight

Lethe wasn’t just a thought experiment—it was wired to run on a live Cognee Cloud tenant. The tool mapped the platform’s memory lifecycle:

• remember → add + cognify
• recall → search
• forget → the dedicated forget endpoint

A live round-trip test confirmed the system’s functionality: add a subject, recall their details (leaks detected), trigger forget, then recall again (silence). But the real discovery came during testing.

Deleting a dataset row alone—using a standard DELETE operation—removed the raw data but left the cognified graph still answering with the subject’s details. True erasure required using the forget endpoint while the dataset was still intact. Lethe’s Auditor made this difference visible, prompting a pipeline fix upstream to align deletion with privacy expectations.

A compliance-ready interface: From proof to product

Lethe’s demo wasn’t just about back-end verification—it was designed to tell a story. The interface includes four core screens:

• Memory graph: A force-directed visualization rendered on plain canvas (no libraries) shows customers, cities, and complaint types with cross-links. Erased subjects collapse into redacted red ghost nodes, visually proving their absence.
• Prove on Cognee Cloud: A live panel executes the Auditor’s probes on the cloud tenant, displaying before-and-after answers with PII highlighted for clarity.
• Compliance cockpit: A dashboard for data protection officers, showing contamination and erasure status for every subject. It also maintains an append-only ledger of signed certificates, exportable as JSON or CSV for regulators.

The lesson: Memory layers should enable verification

The most valuable insight from this project wasn’t about exposing leaks—it was about the power of a well-designed memory abstraction. Cognee’s lifecycle model (remember, recall, improve, forget) made the memory layer boring, allowing the focus to shift to the real challenge: verifying whether the AI truly forgot. When the plumbing is simple, the interesting problems become solvable.

As AI systems grow more pervasive, the demand for provable erasure will only intensify. Tools like Lethe don’t just audit memory—they redefine trust in the age of AI.