Finding a psycopg connection leak in executor threads

A pod started restarting. The autoscaler had decided the database pool was saturated, brought up a replacement, and killed the old one.

I had just raised the pool limit, so I opened Prometheus.

From September 11 the line only went one way. pool_size climbed toward 50, which is pool_max. pool_available sat between 2 and 5 even while the pool grew. That graph is one worker pod.

One worker pod. Purple is pool_size, climbing toward max=50. Yellow is pool_available, stuck around 2-5 even as the pool grows.

Sustained traffic looks like that. A leak looks like that. Both at once also look like that.

close_old_connections() is not putconn()

Most ORM work already went through wrappers that call Django's connection cleanup:

close_old_connections()

With the pooled psycopg backend, that does not close the PostgreSQL socket. It returns the lease to psycopg-pool so another request can use the connection. That is the request-thread contract. It only runs on the thread that checked the connection out.

I found two paths that skipped it. One of them was Bedrock profile provisioning.

Five workers, five leases

An API request created a short-lived ThreadPoolExecutor. Each worker did ORM work: create or check inference-profile rows.

HTTP request
  → ThreadPoolExecutor(max_workers=5)
  → worker ORM query
  → psycopg-pool getconn()
  → worker exits
  → no putconn()

The request thread could call close_old_connections() all it wanted. Those five connections lived on other threads.

GC closed the socket. The pool still counted it out.

When a worker exited, its thread-local Django connection wrapper could stay alive in a reference cycle. Later, garbage collection closed the underlying PostgreSQL socket.

Closing the socket is not putconn(). psycopg-pool still counted that slot as checked out. Postgres had dropped the session. The pool thought it was busy.

That is why pool_available stayed tiny while pool_size marched to 50. The pool was growing to replace connections it believed were still in use.

Reproducing it

I hit the full authenticated HTTP endpoint locally. Each request dropped pool_available by five: one slot per executor worker. Cleanup inside the worker kept availability flat.

OpenSearch already had the Bedrock provisioning logs. The API path had run in production and spawned multiple workers. Those logs proved the path executed. The local full-request repro proved missing cleanup could retain leases.

Make the executor own cleanup

The fix was to stop trusting callers:

class DatabaseThreadPoolExecutor(ThreadPoolExecutor):
    def submit(self, fn, /, *args, **kwargs):
        return super().submit(_run_with_database_connection_cleanup, fn, args, kwargs)
 
 
def _run_with_database_connection_cleanup(fn, args, kwargs):
    close_old_connections()
    try:
        return fn(*args, **kwargs)
    finally:
        close_old_connections()

The Bedrock code now uses this executor. Every submitted task cleans up Django connections on the same thread that acquired them, including the failure path.

After deploy, the same request no longer reduced available pool capacity.