Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 100 additions & 3 deletions README.org
Original file line number Diff line number Diff line change
Expand Up @@ -254,9 +254,15 @@ a TCP handshake or a database login — still needs to happen before the member
is safe to use.

The value is an ={M, F, A}= tuple. Before calling, pooler replaces placeholder
atoms in =A=: ='$pooler_pid'= with the pid of the newly started member, and
='$pooler_pool_name'= with the member supervisor name for the pool (same
placeholders supported by =stop_mfa=):
atoms in =A= with the corresponding runtime values (same placeholders are
supported by =stop_mfa=):

- ='$pooler_pid'= — the pid of the newly started member
- ='$pooler_member_sup'= — the name of the member supervisor that owns this
worker (shard-specific when [[#numbermembersups][=num_member_sups > 1=]])
- ='$pooler_pool'= — the pool name atom
- ='$pooler_pool_name'= — *deprecated* alias for ='$pooler_member_sup'= kept
for backward compatibility; existing configs continue to work unchanged

#+BEGIN_SRC erlang
#{name => pg_pool,
Expand Down Expand Up @@ -295,6 +301,41 @@ The callback is called from the helper process instead of
=supervisor:terminate_child/2=. For maximum throughput under churn, prefer a
fast or fire-and-forget teardown so that slots are freed quickly.

**** Graceful member shutdown (member_shutdown)

By default pooler kills member processes with =brutal_kill= (immediate exit,
no cleanup). Set =member_shutdown= to a millisecond timeout to allow the
member's =terminate/2= callback to run before the supervisor considers it done:

#+BEGIN_SRC erlang
#{name => pg_pool,
...
member_shutdown => 5000} %% wait up to 5 s for clean shutdown
#+END_SRC

The value maps directly to the [[https://www.erlang.org/doc/apps/stdlib/supervisor.html][OTP supervisor child spec]] =Shutdown= field:
=brutal_kill= sends =exit(Pid, kill)= (immediate, the default); a
=pos_integer()= sends =exit(Pid, shutdown)= and waits up to that many
milliseconds.

Two caveats:

- *Requires the worker to trap exits.* =exit(Pid, shutdown)= kills a process
that has not called =process_flag(trap_exit, true)= immediately — including
gen_server-based workers, which do /not/ set trap_exit automatically.
Workers must explicitly call =process_flag(trap_exit, true)= in their
=init/1= for =terminate/2= to run on shutdown.

- *Ignored by custom =stop_mfa= that bypasses the supervisor.* If =stop_mfa=
calls the worker directly (e.g. =epgsql:close/1=) rather than going through
=supervisor:terminate_child/2=, the =Shutdown= field is never consulted.
The default =stop_mfa= uses =supervisor:terminate_child/2= and respects
this setting.

Graceful shutdown interacts well with =num_member_sups=: stops across
different shards proceed in parallel, reducing total teardown time from
O(N × timeout) to O(N × timeout / num_member_sups).

**** Pools with slow-starting or slow-stopping workers

For pools whose workers take significant time to start (e.g. opening a database
Expand All @@ -314,6 +355,55 @@ or network connection), the recommended configuration pattern is:
For pools with workers that are slow to stop, use =stop_mfa= with a fast
asynchronous teardown so that the pool is not blocked while workers drain.

**** Sharded member supervisors (num_member_sups)
<<numbermembersups>>

This feature mirrors the =num_conns_sups= option introduced in
[[https://ninenines.eu/articles/ranch-2.0.0/][Ranch 2.0]] to address the same supervisor mailbox bottleneck in
connection listeners.

By default each pool has a single =simple_one_for_one= supervisor that owns all
member processes. Concurrent =start_child= / =terminate_child= calls (from
parallel starters, cull events, and async stoppers) serialize through that
supervisor's mailbox, so a slow =start_mfa= can stall the pool under load even
when individual workers are fine.

The =num_member_sups= pool configuration option splits ownership across N
parallel supervisors. New starts are distributed round-robin and each worker
remembers which shard owns it, so terminations route to the correct supervisor.
Defaults to =1= (single supervisor, identical to the legacy layout).

#+BEGIN_SRC erlang
#{name => pg_pool,
init_count => 32,
max_count => 128,
start_mfa => {my_conn, start_link, []},
num_member_sups => 8} %% 8 parallel member supervisors
#+END_SRC

When to use it:

- Workloads with non-trivial =start_mfa= cost where =initialize_mfa= is not
an option (e.g. third-party libraries that only expose a monolithic
=start_link/N=). With =initialize_mfa= available, the supervisor is held only
for the cheap =start_link= portion and a single shard is usually sufficient.
- Workloads using a graceful =member_shutdown= timeout: with a single
supervisor all shutdowns serialise, so total stop time is O(N × timeout);
with M shards it becomes O(N × timeout / M).

The cost of extra shards is negligible — each shard is one lightweight
supervisor process (~2–3 KB). When choosing N, the relevant figure is the
expected peak number of *concurrent* starts or stops rather than the total pool
size. For I/O-bound =start_link= (e.g. opening TCP connections), N can safely
exceed the CPU scheduler count since blocked processes yield the scheduler.
=N > max_count= is never useful — at most =max_count= workers can exist at any
moment, so surplus shards sit permanently empty.

=num_member_sups= can be /increased/ via =pooler:pool_reconfigure/2=; existing
members stay on their original shards and new starts fill the added shards
round-robin until churn rebalances things. /Decreasing/ is not supported and
returns ={error, num_member_sups_cannot_be_decreased}=.

*** Pool Configuration via =pooler:new_pool=
You can create pools using =pooler:new_pool/1= when accepts a
map of pool configuration. Here's an example:
Expand All @@ -337,6 +427,8 @@ pooler:pool_reconfigure(rc8081, PoolConfig#{max_count => 10, init_count => 4}).
It will update the pool's state and will start/stop workers if necessary, join/leave group,
reschedule the cull timer etc.
The only parameters that can't be updated are ~name~ and ~start_mfa~.
~num_member_sups~ can only be increased (not decreased) — see
[[#numbermembersups][the sharding section]] for details.

However, updated configuration won't survive pool crash (it will be restarted with old config by
supervisor). But this should not normally happen.
Expand Down Expand Up @@ -598,6 +690,11 @@ to start and supervise the members of this pool. The
pooler_starter_sup is used to start temporary workers used for
managing async member start.

With =num_member_sups > 1= (see [[#numbermembersups][Sharded member supervisors]]), the pool
supervisor starts N member supervisors. Shard 1 keeps the legacy name
=pooler_NAME_member_sup= shown above; additional shards are named
=pooler_NAME_member_sup_2=, =pooler_NAME_member_sup_3=, and so on.

pooler_sup: one_for_one
pooler_NAME_pool_sup: all_for_one
pooler_NAME_member_sup: simple_one_for_one
Expand Down
6 changes: 6 additions & 0 deletions src/pooler.appup.src
Original file line number Diff line number Diff line change
@@ -1,4 +1,10 @@
% -*- mode: erlang -*-
%% When bumping the version (e.g. 1.7.0 → 1.8.0) and adding a `1\\.7\\.0.*' entry:
%% remember to include `{update, pooler_pool_sup, supervisor}'. pool_sup's
%% `init/1' builds child specs dynamically from `num_member_sups', so any change
%% to that logic (or to the MFA / id / count of the children it returns) requires
%% the supervisor to re-evaluate `init/1' on hot upgrade — `{load_module, ...}'
%% alone leaves the running supervisor with stale specs.
{"1.7.0",
[{<<"1\\.6\\.0.*">>,
[{update, pooler, {advanced, []}},
Expand Down
Loading
Loading