Postgres Connection Pool Starvation and Asynchronous Worker Contention: Forensic Patterns and Fixes

One of the most frequent production emergencies we are called to triage is database connection pool starvation. The symptoms are familiar: during an otherwise routine traffic bump, p99 latencies jump from 45ms to 12,000ms, API gateways begin returning 504 Gateway Timeouts, and telemetry dashboards report maxed-out connection pools while CPU utilization on the database instance sits deceptively low at 18%.

The Anti-Pattern: Network I/O Inside Transaction Boundaries

When database CPU is low but connections are exhausted, the bottleneck is almost always transaction duration rather than query execution cost. In 70% of audits, we discover developers wrapping external HTTP calls (such as payment gateway authorizations, email dispatches, or webhooks) inside open database transactions:

BEGIN TRANSACTION;
UPDATE orders SET status = 'processing' WHERE id = 10492;
-- Flaw: 800ms network round-trip holding an active connection
http_response = call_payment_gateway(order.amount);
UPDATE orders SET status = 'paid', payment_ref = http_response.id WHERE id = 10492;
COMMIT;

During normal operations with 5 requests per second, holding a connection open for 800ms goes unnoticed. But when traffic increases to 80 requests per second, the entire pool of 100 connections is consumed within milliseconds, starving all other concurrent queries.

Establishing Transaction Disciplines

Our architectural remediation introduces three rigid invariants:

1. Zero Network Calls in Transactions: No outbound socket operations, file reads, or remote service calls are permitted between BEGIN and COMMIT.
2. Two-Phase State Machine Workflows: Persist state as pending_payment, commit the transaction to release the connection, perform the external HTTP call, and then acquire a second short transaction to record the result.
3. Sizing Pool Limits by CPU Core Allocation: Over-allocating pool connections beyond what the database storage engine can context-switch simultaneously degrades overall throughput. We right-size application pools and introduce connection multiplexers like PgBouncer.