WAL makes memory full due to SAS token expiration

[vishaltanawade]

I am using CNPG 1.26 with Barman plugin 0.11.0 . I am taking backup on azure blob storage. I am using SAS token to connect with azure blob storage . But when SAS token expire and I forgot to update it . after that it looks like it stores wals inside pods memory which increases memory consumption which results it into pod not working. But After I update sas token it should release wal and make memory free. but it did not happen. It deletes the wal only after increasing the db cluster memory . Is there any other way to resolve this issue or release this memory.

@dosu, please help

[dosubot]

This is a known pain point with WAL archiving in CloudNativePG. When your SAS token expires, WAL files accumulate locally in PostgreSQL's pg_wal directory and the spool directory (/controller/wal-archive-spool) because they can't be shipped to Azure Blob Storage. Once you restore valid credentials, PostgreSQL will automatically retry archiving the pending WAL files, but the local cleanup happens only through PostgreSQL's normal checkpoint process—there's no built-in "flush now" command [1].

Here's what you can do without resizing the cluster:

1. Force a WAL switch and checkpoint after updating the SAS token:

   SELECT pg_switch_wal();
   CHECKPOINT;

   This forces the current WAL segment to close and triggers PostgreSQL to clean up WAL files that have been successfully archived [2].

2. Verify archiving is working again by checking for .ready files:

   ls /var/lib/postgresql/data/pg_wal/archive_status/*.ready

   Once these are gone (converted to .done), the archived WALs become eligible for cleanup on the next checkpoint.

3. For future prevention, consider:

   • Using a dedicated WAL volume via .spec.walStorage so WAL accumulation doesn't fill your main data PVC [3]
   • Switching to Azure AD Managed Identity authentication instead of SAS tokens to avoid credential expiration entirely [4]
   • Setting max_slot_wal_keep_size to cap how much WAL is retained:

     postgresql:
       parameters:
         max_slot_wal_keep_size: "2GB"

4. If the pod is OOMKilled or unresponsive, and you can't run SQL commands, you may need to temporarily increase memory as you did, or delete the pod to let it restart—at which point it will attempt to archive pending WALs on startup via the ArchiveAllReadyWALs() function [5].

The key takeaway: after fixing the SAS token, a CHECKPOINT should eventually free the space, but it may take a few checkpoint cycles for PostgreSQL to fully reclaim all the WAL segments. If the pod is already under extreme memory pressure, a restart (pod deletion, not cluster resize) may be needed to recover.

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[vishaltanawade]

in pod i got the below logs

Back-off restarting failed container postgres in pod devl-iee-project-db-271-1_iee(c94fee33-3fa4-4f0f-9763-96fbe326d4f7)
Starting CloudNativePG Instance Manager
Checking for free disk space for WALs before starting PostgreSQL
Detected low-disk space condition, avoid starting the instance

this types of logs i got in postgres container .

i updated sas token , later restarted the pod but still not working .
is there any way except memory increase

@dosu, please help

[dosubot]

The log message Detected low-disk space condition, avoid starting the instance means the instance manager found that your pg_wal directory has less free space than 2 WAL segments (typically 2 × 16MB = 32MB) [1]. This is a hardcoded safety check with no bypass option—no annotation, env var, or config can skip it [2].

Since your SAS token was expired, unarchived WAL files accumulated on disk and filled the volume. Updating the token alone won't help because PostgreSQL can't start to process/archive those WALs—it's a chicken-and-egg problem.

Here are your options without permanently increasing cluster memory/storage:

Option 1: In-place PVC resize (recommended)

If your StorageClass supports volume expansion (allowVolumeExpansion: true), you can temporarily expand the PVC size:

spec:
  storage:
    size: <larger_size>
    resizeInUseVolumes: true

Once PostgreSQL starts and archives the pending WALs, the space will be reclaimed naturally. You could then resize back down on a new cluster if needed [3].

Option 2: Manually free space from inside the pod

Since the pod is running but Postgres won't start, you can exec into the container and manually remove already-archived WAL files. To identify which WALs are safe to remove:

kubectl exec -it <pod-name> -c postgres -- bash
# Check which WALs are pending archival (.ready = not yet archived)
ls /var/lib/postgresql/data/pg_wal/archive_status/*.ready
# Check which WALs are already archived (.done = safe to remove)
ls /var/lib/postgresql/data/pg_wal/archive_status/*.done

You can remove WAL files that have a corresponding .done file in archive_status/, as those have already been archived. Do NOT remove .ready files—those haven't been archived and removing them could cause data loss.

# Example: remove already-archived WAL files
for f in /var/lib/postgresql/data/pg_wal/archive_status/*.done; do
  walfile=$(basename "$f" .done)
  rm -f "/var/lib/postgresql/data/pg_wal/$walfile"
  rm -f "$f"
done

⚠️ Caution: This is not an officially recommended procedure and carries risk. Only remove files with .done status.

Option 3: Clear the spool directory

The plugin also stores WALs in /controller/wal-archive-spool. Check if that's consuming space and clean it:

kubectl exec -it <pod-name> -c postgres -- ls -la /controller/wal-archive-spool/

Going forward, I'd strongly recommend:

• Using a dedicated WAL volume via .spec.walStorage so WAL accumulation doesn't fill your data PVC [4]
• Switching to Azure AD Managed Identity instead of SAS tokens to avoid credential expiration [5]
• Setting up monitoring/alerts on PVC usage to catch this before it becomes critical

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.