fix: garbage-collect the Lua VM before snapshotting

[jim-toth]

Summary

dev_lua:snapshot/3 serializes the whole luerl VM with term_to_binary(luerl:externalize(State)) and never garbage-collects it. Luerl reclaims table-store slots only inside luerl_heap:gc/1, and nothing in HyperBEAM calls it, so the snapshot retains every table the process has ever allocated and grows without bound, even when the Lua-visible state is a fixed size. It is written on every snapshot slot, so per-message cost grows with accumulated slots.

Measurements

Reproduced on edge 7135fdba using test/test.lua as the fixture, with the process driven through its default compute, which writes the same result on every message. Sampling the serialized snapshot as slots accumulate:

after N messages	edge	with luerl:gc
2	61,578 B	21,061 B
25	500,338 B	23,853 B
50	978,383 B	24,163 B
100	1,935,003 B	28,038 B

Stock grows ~19 KB per slot and is 69× larger by slot 100, for a workload whose Lua-visible state does not grow. That growth is dead tables the VM has never been asked to collect. Exact byte counts move by ~0.1% between runs, since each run spawns a process with a fresh identity and those bytes sit inside the serialized state; the ratios are stable.

Collected, the snapshot tracks live state rather than slot count. It is not byte-identical across those samples, and should not be: the state handed to the script carries per-slot process fields, so successive snapshots differ legitimately. Byte-identity is only meaningful where the Lua state between two snapshots is genuinely unchanged and that case is measured below.

Restore

snapshot/3 does not write the collected state back into priv, collects a copy purely for serialization, and the running VM continues on the uncollected state. The only behavioral surface is restore which is measured directly: drive a process 40 messages, snapshot, restore through normalize/3 into a base with no priv state, then re-snapshot and continue.

after 40 messages	edge	with luerl:gc
snapshot	771,017 B	25,893 B
restores via normalize/3	ok	ok
re-snapshot byte-identical	true	true
continues, count 40 → 41	41	41
matches the never-restored process	true	true

The restored VM re-serializes to byte-identical bytes and continues to the same result as the process that never restarted. This is also the unchanged-state case referred to above: between those two snapshots the Lua state really is identical, and the bytes match exactly.

Why this cannot be fixed at the application layer

Per CONTRIBUTING rule 2, the application-layer fix was tried first: calling collectgarbage() from inside the Lua contract. It does not suffice: it is unsound, and it fails inside HyperBEAM.

In luerl 1.3.0, the version pinned in rebar.lock here, collectgarbage("collect") is not a stub: luerl_lib_basic:collectgarbage/3 calls luerl_heap:gc/1 on the in-flight state. luerl_heap:gc/1 derives its root set from the state handed to it, so it is only correct at a settled point, which is what snapshot/3 is. With a frame on the dynamic stack it cannot see the in-flight continuation, and objects still reachable from that frame are freed.

Run through lua@5.3a, a contract calling collectgarbage() inside a pcall:

Erlang error while running Lua: {badkey,29}
  erlang:map_get/[29, ...]
  status 500, phase compute

The Lua pcall does not catch it: the error is raised below Lua, in the Erlang heap code, and the whole computation fails. Identical on unmodified edge, so this is a property of luerl rather than anything this PR changes.

No Lua program can safely collect its own VM, so the collection point has to be one the application layer cannot reach.

Note on luerl's externalize/1

Misleadingly named: luerl_lib_math:externalize/1 converts only the RNG state so it survives term_to_binary. It does not touch the table store, so it is not a substitute for collecting.

Testing

rebar3 device test --with-core (the full suite: every device plus the core modules) against edge at 7135fdba, and against this branch: 3,497 passed, 5 failed, identical both ways. The same 5 fail on unmodified edge: four scheduler@1.0 http_get_legacy_* tests and push@1.0: test_push_prompts_encoding_change, so this PR introduces no new failures or flakes (rule 1). rebar3 device test --devices dev_lua on its own: 38/38, including pure_lua_restore_test.

For completeness: the teardown printed Segmentation fault after All 38 tests passed on this branch and on unmodified edge. It is unrelated to this change and I have not investigated it further.

[samcamwilliams]

Great find! Thanks @jim-toth . The docs for Luerl claim that we should be able to trigger this from inside the Lua env, but I think the point that it would be nice for this to be automatic(!) is sound. The question is how best to avoid OOM, maintain performance, while ensuring that we don't encode the GC mechanics of Luerl into ~lua@5.3a's protocol.

There are a couple of different paths we could go from here:

1. GC on snapshot: If I am not misunderstanding this, I think we would eventually OOM from just 'straight line' execution and then upon recovery from the snapshot we would start again from a GC'd result.
2. GC after compute: Possibly too frequent? But at least ensures robust, deterministic results.
3. GC probabilistically after compute: Fixes the frequency issue with 2. phash2(LuaState rem GCFrequency) would be an obvious way to determine whether or not to GC, but likely too slow?
4. GC upon size increase after compute: Same as 3, but use the size of resulting Luerl environment (relative to its starting size) as the trigger. erlang:external_size or erts_debug:[shared|flat|external_]size would likely be too slow still, but I think we might be able to use the size in memory of the Erlang process itself (measured before and after) to approximate the increase in O(1). It is pretty ... gross/sketchy, though. One crux to be careful of here would be 'boiling frogs'. If we only GC every time the process memory increases by 1% or more (for example), we could accidentally avoid GC'ing for extremely large periods if we get unlucky. Specifics of the workload could compound this 'luck', too.
5. GC on a [possibly deterministic] counter after compute: Add a new priv/ field that tracks the invocations of the Luerl state. We could even just wrap the current Luerl state with this counter, then have 'memory_safe_*' functions for invoking Lua that increment and GC at the right moments?

5 seems cleanest, but open to ideas!
Sam

[jim-toth]

Thanks @samcamwilliams.

I implemented option 5 (counter in priv, collect when count rem N == 0) so that N=1 is option 2 (always collect) on stock edge, and measured it across different processes since the cost depends almost entirely on shape.

Luerl gc is mark-and-sweep with the mark phase being the heavy part, and it scales by number of objects rather than data size, so what a process keeps live matters far more than what it allocates.

what the process keeps live in the VM	live tables	live entries	gc, % of compute	VM after 200 slots uncollected	collected
almost nothing (base.n = base.n + 1)	~0	~0	0.5%	2.3 MB	13 KB
addressable state, on the message not in the VM	~0	~0	0.003%	344 MB	17 KB
~12,000 entries across a few flat maps	4	12,000	8%	3.7 MB	1.2 MB
~24,000 entries across a few flat maps	4	24,000	9%	4.8 MB	2.4 MB
~48,000 entries across a few flat maps	4	48,000	22%	7.2 MB	4.7 MB
~12,000 entries as 4,000 small records	4,001	12,000	95%	3.3 MB	0.9 MB

Rows three and six hold the same 12,000 entries: reshaping them into 4,000 small tables is what takes it from 8% to 95%, and turning GC on there makes the process 16.5x slower.

Cost and benefit run opposite, which is the useful part: the shapes cheapest to collect have the most to reclaim.

Deferring converts the cost to memory rather than avoiding it. At N=25 the second shape carries up to 41.6 MB between collects instead of a flat 17 KB.

So I'd argue for N=1 as the default, with the counter kept as a knob. No single frequency is right for every shape, but the safe one should be what you get without asking for it.

I would put lua-gc-frequency in the process definition rather than node config, so every node computing a process collects at the same points by construction.

0 meaning never is worth having as an escape hatch. Defaulting to 0 would keep existing processes bit-identical, but it leaves the leak as the default and only fixes it for people who already know about it.

Either shape replaces the snapshot patch in this PR, since collecting in process_response means the state is already collected by the time anything serializes it. Worth noting luerl_emul:call/3 already carries %% Should do GC here. internally, so collecting between calls looks like where rvirding expected it.

Would you rather this PR is repurposed or a fresh one opened with this new proposed change?

Re: GC from Lua

One correction to what I posted earlier: collectgarbage() does work from Lua, but only from the outermost frame, and calling it inside a pcall throws an error Lua can't catch. So a process can bound its own growth, but only while compute is its outermost frame, which it has no way to check.