PostgreSQL HA dashboards: catch hidden replication failures early

High availability (HA) is often treated as a binary state: you either have a replica or you don’t. In PostgreSQL, however, a replica can appear healthy in monitoring while silently falling behind or disconnecting for days—only to reveal itself as a single-node system when the primary node fails. The difference between "having a replica" and "being truly highly available" lies in the visibility of replication status.

A robust HA dashboard for PostgreSQL streaming replication doesn’t just track whether a standby exists; it surfaces lag at every stage of the WAL journey—from primary outbox to standby replay—and triggers alerts before replication gaps become production incidents.

How PostgreSQL streaming replication works under the hood

In a PostgreSQL HA setup, the primary node runs a walsender process for each standby. Each walsender reads Write-Ahead Log (WAL) records from the primary’s pg_wal directory (or from memory buffers if the data is still hot) and streams them over a TCP connection to the standby. On the standby side, a walreceiver process receives each WAL record, writes it to the local pg_wal directory, optionally performs an fsync (depending on synchronous_commit settings), and then the startup process applies the record into shared buffers—this final step is called replay.

At each step, four key Log Sequence Numbers (LSNs) are tracked:

• sent_lsn — the last byte the primary sent over the socket to the standby.
• write_lsn — the last byte the standby wrote into the operating system’s page cache.
• flush_lsn — the last byte the standby confirmed as written to disk via fsync.
• replay_lsn — the last byte replayed into shared buffers (meaning the data is now visible on the standby).

According to PostgreSQL documentation, these values must always satisfy: replay_lsn <= flush_lsn <= write_lsn <= sent_lsn <= pg_current_wal_lsn()

The gaps between these points represent lag. PostgreSQL also computes time-based lag metrics—write_lag, flush_lag, and replay_lag—as intervals between when a WAL record is generated on the primary and when it completes each step on the standby. These are derived from periodic feedback messages sent from the standby back to the primary.

Essential queries for a real-time HA dashboard

To build a dashboard that reflects true HA health, you need to query three PostgreSQL system views on the primary:

SELECT
  application_name,
  client_addr,
  state,
  sync_state,
  pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn) AS sent_lag_bytes,
  pg_wal_lsn_diff(pg_current_wal_lsn(), write_lsn) AS write_lag_bytes,
  pg_wal_lsn_diff(pg_current_wal_lsn(), flush_lsn) AS flush_lag_bytes,
  pg_wal_lsn_diff(pg_current_wal_lsn(), replay_lsn) AS replay_lag_bytes,
  write_lag,
  flush_lag,
  replay_lag,
  EXTRACT(EPOCH FROM (now() - reply_time)) AS seconds_since_reply
FROM pg_stat_replication
WHERE state = 'streaming';

On the standby, pg_stat_wal_receiver provides the heartbeat of the replication process:

SELECT
  conninfo,
  sender_host,
  status,
  last_msg_send_time,
  last_msg_receipt_time,
  latest_end_lsn
FROM pg_stat_wal_receiver;

This view is critical: if no row exists, the standby is disconnected or not yet reconnected. Two functions help measure lag on the standby:

• pg_last_wal_replay_lsn() — the last LSN replayed.
• pg_last_xact_replay_timestamp() — the timestamp of the last committed transaction on the standby.

The difference between now() and pg_last_xact_replay_timestamp() gives you the time lag in seconds—how far behind the standby is compared to the latest commit on the primary.

Replication slots add another layer of safety. A physical replication slot pins WAL segments so the primary doesn’t recycle them before the standby consumes them. The pg_replication_slots view tracks slot status:

SELECT
  slot_name,
  slot_type,
  active,
  wal_status,
  pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
  pg_size_pretty(safe_wal_size) AS safe_wal_remaining
FROM pg_replication_slots;

The wal_status column (introduced in PostgreSQL 13) can be reserved, extended, unreserved, or lost. A lost status means the WAL has been recycled and the standby must resync from a base backup. The safe_wal_size tells you how many bytes of WAL remain before the slot is at risk of being lost.

Three silent failure modes that break HA in production

Replication can fail in ways that evade basic monitoring. Here are the most common scenarios teams miss until it’s too late.

1. Standbys disconnect without triggering alerts

A standby can stop receiving WAL for hours without any visible alert. Common causes include:

• TCP connection drops due to firewall idle-timeouts or network blips.
• The wal_receiver process crashing on the standby.
• wal_receiver_timeout expiring during a network delay.
• Misconfigured primary_conninfo after a standby restart.

The primary’s pg_stat_replication view simply stops showing that standby. There’s no exception thrown; the log may only show a brief line like:

terminating walsender process due to replication timeout

A naive alert like SELECT count(*) > 0 FROM pg_stat_replication fails because it only checks total replica count, not individual health. You need to monitor each standby by name using application_name, set during connection setup via:

primary_conninfo = 'host=primary user=repl password=*** application_name=standby-prod-1'

Then alert using:

SELECT
  expected.name,
  CASE 
    WHEN r.application_name IS NULL THEN 'DOWN'
    WHEN r.state <> 'streaming' THEN r.state
    ELSE 'ok'
  END AS health
FROM (VALUES ('standby-prod-1'), ('standby-prod-2')) AS expected(name)
LEFT JOIN pg_stat_replication r ON r.application_name = expected.name;

Without application_name, you cannot identify which replica failed—making incident response guesswork.

2. Stale replication slots fill up disk space on primary

When a standby disconnects but the replication slot (primary_slot_name) remains active, the slot’s restart_lsn does not advance. The primary keeps all WAL segments from restart_lsn onward indefinitely. This causes pg_wal/ to grow without bound until the disk is full, triggering a PANIC: could not write to file ... No space left on device error.

This exact scenario contributed to a major outage at GitLab in 2017, where a missing monitoring signal allowed WAL accumulation to go undetected. PostgreSQL 13 introduced max_slot_wal_keep_size to cap this growth, but you still need an alert on wal_status IN ('extended','lost') to act before the slot is lost and requires a full resync.

3. Synchronous replication deadlocks when sync standbys vanish

When synchronous_commit = 'on' and one or more sync standbys disappear, the primary blocks all write transactions until a sync standby reconnects or synchronous_standby_names is updated. This can freeze the entire database cluster, even if async standbys are healthy.

Teams often overlook this dependency because monitoring focuses on async replication lag rather than synchronous state. A dashboard must include sync_state from pg_stat_replication and alert when sync standbys drop off.

Build a proactive HA monitoring strategy

A true HA dashboard isn’t just a replica counter—it’s a live status board that surfaces every critical signal:

• Individual standby health by application_name.
• Real-time lag in bytes and time across all stages.
• Standby heartbeat freshness (seconds_since_reply).
• Replication slot status and WAL retention.
• Synchronous standby presence and state.

Start with these queries, integrate them into your observability stack, and set up alerts that wake someone up before replication fails—not after the primary node dies. True high availability isn’t about having a replica; it’s about knowing it’s alive, fast, and reliable.

The next time your primary fails, you’ll know exactly whether you’re truly protected—or just pretending.