Why missing data isn’t always lost: The 16-minute sync gap lesson

When a client reported a critical discrepancy in their daily banking return report, the numbers didn’t lie: 423 lines in the emailed version versus 1,351 in the on-demand portal. The initial assumption? Data corruption. The reality? A 16-minute timing gap.

The illusion of missing data

The client’s frustration was understandable. A report that should have mirrored the same dataset across two delivery methods showed a stark difference. The emailed report, generated automatically at 7:15 AM, captured only a fraction of the expected data. Meanwhile, the portal—accessed later—displayed the full dataset, including the missing lines. The key insight? The data wasn’t lost—it simply hadn’t arrived yet.

Narrowing the gap between assumptions and evidence

Debugging began not with code, but with verification. A manual comparison of both files revealed that every line in the emailed report existed in the portal’s dataset. The discrepancy wasn’t a data error; it was a timing issue. The next step was to systematically eliminate plausible causes:

• Data splitting across systems? No. All 1,351 lines belonged to the same bank code.
• Overly aggressive deduplication? A test query replicating the production logic confirmed no rows were removed.
• Inconsistent date filters? Both reports used the same transaction_created_at column for the same date.

Each hypothesis was dismantled with read-only queries, leaving only one explanation: the data hadn’t been fully imported when the report ran.

The smoking gun: timestamps don’t lie

The breakthrough came from tracing the timestamps of two critical processes:

• The report job ran at 7:15:45 AM, generating the emailed output.
• The bank’s return file, containing ~12,700 transactions for the day, was imported at 7:31:02 AM.

The 16-minute gap between these events explained the missing lines. A breakdown of the data by this cutoff confirmed the pattern:

• Invoices imported before 7:15 AM: 423 lines (captured by the email report)
• Invoices imported after 7:15 AM: 928 lines (missing from the email report, visible in the portal later)
• Total invoices: 1,351

The portal’s report “saw more” simply because it was accessed after the import completed at 8:23 AM.

Why timing bugs slip through the cracks

This wasn’t a logic error in the report’s SQL query. It was a temporal coupling flaw—a mismatch between when a scheduled task runs and when the underlying data becomes available. The report assumed that all payment confirmations from the previous day would be fully loaded by 7:15 AM, but the bank’s feed arrived 16 minutes later. This is a classic example of a race condition in system design.

Fixing the root cause, not the symptom

Three potential solutions emerged, ranked by practicality and effectiveness:

1. Delay the report until after the import completes – The simplest fix, but shifts the burden to human scheduling.
2. Trigger the report dynamically after the import finishes – A more robust approach that eliminates reliance on fixed timers.
3. Add a pre-flight validation step – The report checks whether the expected import has completed before running, preventing false negatives.

The best solution depends on the system’s tolerance for delay. For a financial report, even a 16-minute lag could have real consequences if it leads to incorrect reconciliations or delayed decision-making.

A universal lesson: timing matters more than you think

Before diving into code or logs, ask two fundamental questions:

• When is this data actually available?
• When am I reading it?

Most “missing data” bugs aren’t about data loss—they’re about reading stale data at the wrong moment. This case underscores a broader truth in system design: synchronization windows are silent time bombs. They don’t always cause crashes, but they erode trust in your data pipeline over time.

The next time you see a discrepancy in your reports, resist the urge to debug immediately. Start by mapping the lifecycle of your data from creation to consumption. You might find the bug isn’t in the code—it’s in the clock.