From 031a21b9283a24ba377672051dc0ac2e445a194e Mon Sep 17 00:00:00 2001 From: ascottDI Date: Tue, 28 Jul 2026 16:19:01 +0100 Subject: [PATCH 01/10] initial creation of servers module plus refinements --- di/servers/init.q | 3 + di/servers/servers.md | 124 +++++++++++++++++++++++ di/servers/servers.q | 231 ++++++++++++++++++++++++++++++++++++++++++ di/servers/test.csv | 31 ++++++ 4 files changed, 389 insertions(+) create mode 100644 di/servers/init.q create mode 100644 di/servers/servers.md create mode 100644 di/servers/servers.q create mode 100644 di/servers/test.csv diff --git a/di/servers/init.q b/di/servers/init.q new file mode 100644 index 00000000..97ceb222 --- /dev/null +++ b/di/servers/init.q @@ -0,0 +1,3 @@ +/ connection management and handle-by-type lookup for the modular torq world. +\l ::servers.q +export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta]) diff --git a/di/servers/servers.md b/di/servers/servers.md new file mode 100644 index 00000000..b1698e43 --- /dev/null +++ b/di/servers/servers.md @@ -0,0 +1,124 @@ +# di.servers + +Connection management and handle-by-type lookup for the modular TorQ world — the `di.*` +analogue of TorQ's `.servers` (`code/handlers/trackservers.q` + `servers.q`), scoped down +for v1: no discovery service, no password/access-list files, no non-TorQ process tracking, +no FinSpace. `process.csv` here is a static **phone book** (who to *dial*), **not** an +identity source — self-identity comes from config, injected by `di.torq`. + +FRAMEWORK-tier module: no hard `di.*` dependencies; `log`, `timer` and `handlers` are all +**injected** (all required, no fallback). + +## init and config + +Standard **one-arg `init[deps]`**: `di.torq` merges this process's resolved config slice into +the same `deps` dict it passes the injectables in, so `deps` carries both the injectable +dependencies **and** the config keys. `init` wires the deps, records self-identity, and +installs two one-time process-global side effects — a `.z.pc` cleanup handler and a 10s retry +timer job. It is **idempotent** (guarded by an internal `registered` flag): `di.torq` calls +it once per process, but a second call refreshes the dep refs without re-registering (a +duplicate `di.timer.addjob` id would throw). `init` does **not** open connections. + +`deps` keys: + +| key | kind | meaning | +|---|---|---| +| `log` | injectable | binary `` `info`warn`error `` `{[c;m]}` logger dict | +| `timer` | injectable | di.timer contract; `addjob` = the 6-arg `custom` form `{[id;func;params;period;mode;opts]}` | +| `handlers` | injectable | di.handlers contract; `register[event;phase;nm;pri;func]` | +| `proctype`/`procname` | config | this process's own identity (required); used to exclude self from `process.csv` | +| `connections` | config | proctypes this process should dial (symbols, or strings from a `.toml` cascade — normalised). Optional; default = none | +| `processcsv` | config | **path** to `process.csv`; supplied by di.torq. Optional; required only once `connections` is non-empty | + +```q +svc:use`di.servers +svc.init[deps] / deps = injectables + config, assembled by di.torq +svc.startup[] / open the configured connections (reads init config) +h:svc.gethandlebytype[`hdb;`any] +h "1+1" +``` + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. | +| `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | +| `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | +| `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | +| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per export, for `di.torq` to register with `di.api`. | + +Export is deliberately conservative — only functions `di.torq` or a consumer actually calls +(so `di.api` lists exactly these). The rest are **internal**: `retry` (the scheduled +`serversretry` job — passed to the timer *by value* at init, so it needs no export; it first +runs `cleanup` to sweep ungracefully-vanished handles, then reopens every dead handle), +`cleanup`, `formathp`, `opencon`, `readprocesscsv`, `retryrows`, `selector`, `updatestats`, +`signalfound`, `raiseerror`; plus state (`SERVERS`, `self`, `registered`, `HOPENTIMEOUT`, +`connections`, `processcsv`). + +## The `SERVERS` table + +```q +SERVERS:([]procname:`symbol$();proctype:`symbol$();hpup:`symbol$();w:`int$();hits:`int$();startp:`timestamp$();lastp:`timestamp$();endp:`timestamp$()) +``` + +A direct analogue of legacy TorQ's `.servers.SERVERS`: `w` is the live handle (`0Ni` when +disconnected), `hits`/`lastp` drive handle selection, `startp`/`endp` track lifecycle. + +## `.z.pc` registration via di.handlers + +`.z.pc` (connection closed) is a **simple/observer** event in di.handlers — side-effect only, +fan-out — so di.servers registers its cleanup callback through the injected `handlers` +dependency rather than assigning `.z.pc` directly: + +```q +(handlers[`register])[`.z.pc;`;`servers;0j;pcfunc] +``` + +`register`'s signature is `register[event;phase;nm;pri;func]`; for a simple event the `phase` +must be `` ` `` (null) — di.handlers rejects a non-null phase on an observer event. This lets +di.servers' disconnect hook coexist with every other `.z.pc` registrant in the same +priority-ordered fan-out. + +## Conventions (learnings from di.config) + +- **One-arg `init[deps]`** with config folded into `deps` (the project convention; matches + `di.eodtime`'s optional-config-in-deps pattern), not a two-arg `init[config;deps]`. +- **Three-flat-var logging** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, matching + `consistency.md`, `di.compression` and `di.config`. (The project hasn't globally frozen this + vs. the single-dict form — flag before changing.) +- **`raiseerror` (log-then-signal)** for all post-init domain errors (`formathp` unknown + ipctype, `selector` unknown selection, missing `process.csv`). `init`'s own dependency + validation is the one exception (plain `'` — no logger yet). +- **`getapimeta`** exported; a test asserts it documents exactly the module's exports. No + `version` export / VERSION file yet — deferred to the di.depcheck rollout, as in di.config. +- **Env-free** — di.servers reads no environment variable; the `process.csv` path arrives via + `config`processcsv` (di.torq resolves it), holding di.config's env-free boundary. + +## Open items / not yet done + +- **Live-peer integration tests are the next step.** `test.csv` currently covers the mockable + surface (init validation, dep-wiring via recording mocks, idempotency, a no-op `startup`, + `getservers`/`gethandlebytype` with no servers, `getapimeta`). The connection behaviour — + `startup` connecting to a real peer and logging a failed dial, `gethandlebytype` returning a + live remote handle, the retry cycle recovering an ungraceful kill (`cleanup`+reopen), + `waitfortype` connected-vs-timeout — needs a genuinely separate spawned `q` peer (a + self-connect returns pseudo-handle `0`, not a real socket). Since `retry`/`cleanup`/`formathp` + are internal, exercise the retry cycle by invoking the callback the **mock timer captured** at + `addjob` (tests the actually-wired path), not a direct export. See the child-q spawn recipe in + the q-gotchas reference. +- **Provider modules not in kdbx-modules yet.** `di.log` (feature-logging branch) and + `di.handlers` aren't here yet, so the injected contracts are mocked in tests. The handlers + mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`. +- **`config`processcsv` and the assembled `connections` list** depend on di.torq's config + wiring — coordinate when di.torq's servers dep is built. +- Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and the + `tcps`/`unix` socket types end-to-end (only `tcp` is wired through `startup`). + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.servers +``` diff --git a/di/servers/servers.q b/di/servers/servers.q new file mode 100644 index 00000000..450bfb9d --- /dev/null +++ b/di/servers/servers.q @@ -0,0 +1,231 @@ +/ connection management and handle-by-type lookup for the modular torq world - the di.* analogue +/ of TorQ's .servers (code/handlers/trackservers.q + servers.q), scoped down for v1: no discovery +/ service, no password/access-list files, no non-torq process tracking, no FinSpace. process.csv +/ is a static phone book (who to dial), NOT an identity source - self-identity comes from config. +/ FRAMEWORK-tier module: no hard di.* deps; log, timer and handlers are injected (all required). +/ standard one-arg init[deps]: di.torq merges this process's config slice (proctype/procname, +/ connections, processcsv) into the same deps dict it passes the injectables in. conventions match +/ di.config: strict init validation (no fallback), three-flat-var logging, log-then-signal via +/ raiseerror, getapimeta for di.api, and the env-free boundary (the process.csv path arrives via +/ config; di.servers reads no env itself). + +/ --- module-local state (initial values at load; read/written via .z.m at runtime) --- + +SERVERS:([] + procname:`symbol$(); + proctype:`symbol$(); + hpup:`symbol$(); + w:`int$(); + hits:`int$(); + startp:`timestamp$(); + lastp:`timestamp$(); + endp:`timestamp$()); + +HOPENTIMEOUT:2000; + +self:`proctype`procname!``; + +/ guards init's one-time process-global side effects (the .z.pc observer + the retry timer job) so +/ init is IDEMPOTENT - di.torq calls it once per process, but a second call (a test re-run, a +/ future re-init) must not re-register: di.timer.addjob throws on a duplicate id. the dep refs are +/ always refreshed; only the one-time registrations are guarded. +registered:0b; + +raiseerror:{[ctx;msg] + / internal - log an error under ctx via the injected logger, then signal it, so a failure is + / observable in the log as well as thrown. used for all post-init domain errors (init's own + / dependency validation signals with a plain ' - the logger is not wired yet). + .z.m.logerr[ctx;msg]; + '"di.servers: ",string[ctx],": ",msg; + }; + +init:{[deps] + / wire the injected deps (log/timer/handlers - all required, no fallback) and this process's + / config (proctype/procname identity, connections, processcsv), and install the one-time side + / effects (a .z.pc cleanup observer via handlers + a 10s serversretry job via timer). config + / arrives in the SAME deps dict (the one-arg init convention - di.torq merges the config slice + / into it). idempotent (see `registered). does NOT open connections - that is startup's job. + if[99h<>type deps; + '"di.servers: deps must be a dict of injectables + config"]; + if[not all `log`timer`handlers in key deps; + '"di.servers: log, timer and handlers dependencies are required (see di.log, di.timer, di.handlers)"]; + if[99h<>type deps`log; + '"di.servers: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.servers: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + if[99h<>type deps`timer; + '"di.servers: timer value must be a dict (see di.timer)"]; + if[99h<>type deps`handlers; + '"di.servers: handlers value must be a dict (see di.handlers)"]; + if[not all `proctype`procname in key deps; + '"di.servers: proctype and procname (self-identity) are required in deps"]; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; + .z.m.timer:deps`timer; + .z.m.handlers:deps`handlers; + .z.m.self:`proctype`procname!deps`proctype`procname; + .z.m.connections:$[`connections in key deps;deps`connections;`symbol$()]; + .z.m.processcsv:$[`processcsv in key deps;deps`processcsv;""]; + if[not .z.m.registered; + / .z.pc is a SIMPLE (observer) event in di.handlers - side-effect only, fan-out. registered via + / the injected handlers dep with di.handlers' register[event;phase;nm;pri;func] contract; phase + / is ` (null) for a simple event, pri 0. the callback marks a closed handle's row disconnected. + / (param `wh`, not `w`, so it does not shadow the SERVERS column w.) + pcfunc:{[wh] .z.m.SERVERS:update endp:.z.p,w:0Ni from .z.m.SERVERS where w=wh; }; + (.z.m.handlers[`register])[`.z.pc;`;`servers;0j;pcfunc]; + / di.timer mode-1h period is in SECONDS, so 10 = a 10-second retry (a bare 10000 here would be + / ~2.8h - the latent typo that made dead-handle recovery effectively never fire in early POCs). + (.z.m.timer[`addjob])[`serversretry;retry;();10;1;()!()]; + .z.m.registered:1b; + ]; + .z.m.loginfo[`init;"di.servers initialised"]; + }; + +formathp:{[host;port;ipctype] + / internal - build a connection-handle symbol for `tcp`/`tcps`/`unix. only `tcp is exercised by + / startup in v1; the others exist for a future SOCKETTYPE-style config. + h:string $[host=`localhost;`localhost;host]; + p:string port; + $[ipctype=`tcp; lower `$":",h,":",p; + ipctype=`tcps;lower `$":tcps://",h,":",p; + ipctype=`unix;lower `$":unix://",p; + raiseerror[`formathp;"unknown ipctype ",string ipctype]] + }; + +opencon:{[hpup] + / open a connection, logging (not erroring) on failure - a downed peer isn't necessarily an + / error at connect time; retry keeps trying. NOTE the timeout form is hopen[(handle;timeoutms)] + / (a single 2-item list), not the dyadic hopen[handle;timeoutms], which throws 'rank. + r:@[{(hopen (x;.z.m.HOPENTIMEOUT);"")};hpup;{(0Ni;x)}]; + if[null first r;.z.m.logwarn[`servers;"failed to open connection to ",(string hpup),": ",last r]]; + first r + }; + +readprocesscsv:{[path] + / internal - read the static process.csv phone book (host,port,proctype,procname). the PATH is + / supplied by the caller (from config`processcsv); di.servers reads no env itself, holding + / di.config's env-free boundary - di.torq resolves the path and puts it in config. + fsym:`$":",path; + if[0=count key fsym;raiseerror[`readprocesscsv;"process.csv not found at ",path]]; + ("SISS";enlist",") 0: fsym + }; + +startup:{[] + / open connections to every process.csv row whose proctype is in the configured connections list, + / excluding this process's own row. reads the config stored at init. a failed connection is logged + / (not raised) and left as w:0Ni for retry. a no-op if no connections are configured. + / normalise connections to symbols to match process.csv's `proctype column (always a symbol via + / the "S" spec): a .q settings file gives symbols already (`$ throws 'type on a symbol - it is + / NOT idempotent, hence the type check); a .toml one gives plain strings (TOML has no symbol). + conns:.z.m.connections; + conns:$[11h=abs type conns;conns;`$conns]; + if[0=count conns;.z.m.loginfo[`servers;"no configured connections to make"];:()]; + if[0=count .z.m.processcsv;raiseerror[`startup;"processcsv (path to process.csv) is required in config to open connections"]]; + procs:readprocesscsv[.z.m.processcsv]; + pt:.z.m.self`proctype; + pn:.z.m.self`procname; + procs:update isme:(proctype=pt)&procname=pn from procs; + procs:select from procs where not isme; + procs:select from procs where proctype in conns; + if[0=count procs;.z.m.loginfo[`servers;"no process.csv rows match the configured connections"];:()]; + {[row] + hpup:formathp[row`host;row`port;`tcp]; + w:opencon[hpup]; + if[not null w;.z.m.loginfo[`servers;"connected to ",(string row`proctype),"/",(string row`procname)," at ",string hpup]]; + / catenate+reassign, NOT `tablename insert - a symbol-based insert into `.z.m.SERVERS` misses + / the compile-time module-local rewrite a source-level .z.m.SERVERS gets, silently targeting the + / wrong (literal) table. + newrow:([]procname:enlist row`procname;proctype:enlist row`proctype;hpup:enlist hpup;w:enlist w;hits:enlist 0i;startp:enlist $[null w;0Np;.z.p];lastp:enlist .z.p;endp:enlist 0Np); + .z.m.SERVERS:.z.m.SERVERS,newrow; + } each 0!procs; + }; + +retryrows:{[rows] + / internal - reattempt opencon for the given SERVERS row indices, updating w/lastp (and startp on + / a successful reconnect). + hs:opencon each exec hpup from .z.m.SERVERS where i in rows; + .z.m.SERVERS:update w:hs,lastp:.z.p from .z.m.SERVERS where i in rows; + .z.m.SERVERS:update startp:.z.p from .z.m.SERVERS where i in rows, not null w; + }; + +cleanup:{[] + / internal - sweep any row whose handle has vanished from key .z.W (a peer that died WITHOUT a + / clean .z.pc on this side) and mark it disconnected, so retry will reopen it. the .z.pc observer + / already catches clean closes; this catches the ungraceful ones. + dead:exec w from .z.m.SERVERS where not null w, not w in key .z.W; + if[count dead;.z.m.SERVERS:update endp:.z.p,w:0Ni from .z.m.SERVERS where w in dead]; + }; + +retry:{[] + / internal - the scheduled `serversretry job (driven by the injected timer; passed by value at + / init, so it needs no export). first sweep ungracefully-vanished handles (cleanup), then reopen + / every dead (null) handle - so both clean and unclean drops are recovered on the retry cycle. + cleanup[]; + rows:exec i from .z.m.SERVERS where null w; + if[count rows;retryrows[rows]]; + }; + +getservers:{[pt] + / every live (non-null handle) SERVERS row for a proctype. + select from .z.m.SERVERS where proctype=pt, not null w + }; + +selector:{[tab;selection] + / internal - pick one row from a live-server table by algorithm. + $[selection=`roundrobin;first `lastp xasc tab; + selection=`any; rand tab; + selection=`last; last `lastp xasc tab; + raiseerror[`selector;"unknown selection type ",string selection]] + }; + +updatestats:{[wh] + / internal - bump hits/lastp on the row whose handle was just handed out. + .z.m.SERVERS:update lastp:.z.p,hits:1+hits from .z.m.SERVERS where w=wh + }; + +gethandlebytype:{[pt;selection] + / get a single live handle for a proctype via a selection algorithm (`any`roundrobin`last), or + / 0Ni if none is connected. bumps usage stats on the chosen row. + r:getservers[pt]; + if[0=count r;:0Ni]; + wh:(selector[r;selection])`w; + updatestats[wh]; + wh + }; + +signalfound:{[pt] + / internal - log and return 1b once a connection to pt exists. + .z.m.loginfo[`servers;"connected to ",string pt]; + 1b + }; + +waitfortype:{[pt;timeoutms;pollms] + / block until at least one LIVE connection to pt exists, or timeoutms elapses. the DI-scoped + / analogue of legacy TorQ's startupdepcycles - "fail fast, but wait for a hard dependency to come + / up". startup must have run first (so a pt row exists to reattempt). polls retry between tries, + / sleeping pollms. returns 1b once connected, 0b on timeout - the CALLER decides if that is fatal. + / NOTE the blocking system"sleep" is fine at startup (single-threaded; the injected timer's .z.ts + / just doesn't fire during the sleep). + deadline:.z.p+`timespan$1000000*`long$timeoutms; + .z.m.loginfo[`servers;"waiting up to ",(string timeoutms),"ms for a ",(string pt)," connection"]; + while[(0=count getservers pt) and .z.p Date: Thu, 30 Jul 2026 09:32:27 +0100 Subject: [PATCH 02/10] Adding in temporary intergration tests until actual intergration available --- di/servers/servers.md | 22 +++++++------- di/servers/servers.q | 6 ++++ di/servers/test.csv | 50 ++++++++++++++++++++----------- di/servers/test.q | 70 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 121 insertions(+), 27 deletions(-) create mode 100644 di/servers/test.q diff --git a/di/servers/servers.md b/di/servers/servers.md index b1698e43..b85c3248 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -98,16 +98,14 @@ priority-ordered fan-out. ## Open items / not yet done -- **Live-peer integration tests are the next step.** `test.csv` currently covers the mockable - surface (init validation, dep-wiring via recording mocks, idempotency, a no-op `startup`, - `getservers`/`gethandlebytype` with no servers, `getapimeta`). The connection behaviour — - `startup` connecting to a real peer and logging a failed dial, `gethandlebytype` returning a - live remote handle, the retry cycle recovering an ungraceful kill (`cleanup`+reopen), - `waitfortype` connected-vs-timeout — needs a genuinely separate spawned `q` peer (a - self-connect returns pseudo-handle `0`, not a real socket). Since `retry`/`cleanup`/`formathp` - are internal, exercise the retry cycle by invoking the callback the **mock timer captured** at - `addjob` (tests the actually-wired path), not a direct export. See the child-q spawn recipe in - the q-gotchas reference. +- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 33 checks). They spawn a + genuinely separate `q` peer (a self-connect returns pseudo-handle `0`, not a real socket) and + cover: `startup` connecting to a live peer and logging a failed dial while excluding self, + `gethandlebytype` returning a live remote handle (`2=h"1+1"`), the retry cycle recovering an + ungraceful kill (`cleanup`+reopen), and `waitfortype` connected-vs-timeout — plus the mockable + surface (init validation, dep-wiring, idempotency, input validation, `getapimeta`). Because + `retry`/`cleanup` are internal, the retry cycle is driven by invoking the callback the **mock + timer captured** at `addjob` (the actually-wired path), not a direct export. - **Provider modules not in kdbx-modules yet.** `di.log` (feature-logging branch) and `di.handlers` aren't here yet, so the injected contracts are mocked in tests. The handlers mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`. @@ -118,6 +116,10 @@ priority-ordered fan-out. ## Tests +Run in a fresh q session (spawns and kills a real peer process; don't interleave with other +modules' tests). Needs `QHOME` set (the peer is launched via `$QHOME/bin/q`) and `di.os` on +`QPATH` (the harness uses `os.abspath` to load `test.q`): + ```q k4unit:use`di.k4unit k4unit.moduletest`di.servers diff --git a/di/servers/servers.q b/di/servers/servers.q index 450bfb9d..898b4fda 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -168,6 +168,7 @@ retry:{[] getservers:{[pt] / every live (non-null handle) SERVERS row for a proctype. + if[not -11h=type pt;raiseerror[`getservers;"proctype must be a symbol"]]; select from .z.m.SERVERS where proctype=pt, not null w }; @@ -187,6 +188,8 @@ updatestats:{[wh] gethandlebytype:{[pt;selection] / get a single live handle for a proctype via a selection algorithm (`any`roundrobin`last), or / 0Ni if none is connected. bumps usage stats on the chosen row. + if[not -11h=type pt;raiseerror[`gethandlebytype;"proctype must be a symbol"]]; + if[not -11h=type selection;raiseerror[`gethandlebytype;"selection must be a symbol (`any`roundrobin`last)"]]; r:getservers[pt]; if[0=count r;:0Ni]; wh:(selector[r;selection])`w; @@ -207,6 +210,9 @@ waitfortype:{[pt;timeoutms;pollms] / sleeping pollms. returns 1b once connected, 0b on timeout - the CALLER decides if that is fatal. / NOTE the blocking system"sleep" is fine at startup (single-threaded; the injected timer's .z.ts / just doesn't fire during the sleep). + if[not -11h=type pt;raiseerror[`waitfortype;"proctype must be a symbol"]]; + if[not (abs type timeoutms) within 5 7h;raiseerror[`waitfortype;"timeoutms must be an integer (ms)"]]; + if[not (abs type pollms) within 5 7h;raiseerror[`waitfortype;"pollms must be an integer (ms)"]]; deadline:.z.p+`timespan$1000000*`long$timeoutms; .z.m.loginfo[`servers;"waiting up to ",(string timeoutms),"ms for a ",(string pt)," connection"]; while[(0=count getservers pt) and .z.p +/ cleanup, tested in isolation from the auto .z.pc hook (which di.handlers would install for real). +handlercalls:([]event:`symbol$();name:`symbol$()); +mockhandlers:`register`remove`list!( + {[ev;ph;nm;pri;fn]`handlercalls upsert(ev;nm)}; + {[ev;ph;nm]}; + {[ev]}); + +warnlogged:{[s] any (exec msg from logrows where lvl=`warn) like "*",s,"*"}; +firejob:{[id] timerjobs[id][]}; + +/ --- real peer process fixture --- +FIXDIR:"/tmp/diserverstest"; +isfree:{[p] not @[{hclose hopen x;1b};(`$":localhost:",string p;100);0b]}; +pickport:{[start] first (start+til 500) where isfree each start+til 500}; +PEERPORT:0N; DEADPORT:0N; PEERPID:0N; + +waitlisten:{[port;timeoutms] + deadline:.z.p+`timespan$1000000*timeoutms; + while[(.z.p/dev/null 2>&1 &"; + if[not waitlisten[PEERPORT;3000];'"test: peer failed to listen on ",string PEERPORT]; + h:hopen (`$":localhost:",string PEERPORT;2000); + PEERPID::h ".z.i"; + hclose h;}; + +killpeer:{[] if[not null PEERPID;@[system;"kill ",string PEERPID;{}]]; PEERPID::0N; system "sleep 0.3";}; + +setupfixture:{[] + / pick two free ports (peer + a never-listening dead one), then write a header'd process.csv + / phone book with self, the peer (otherproc), and a dead proctype. + PEERPORT::pickport 20000+`int$.z.i mod 20000; + DEADPORT::pickport PEERPORT+1; + system "mkdir -p ",FIXDIR; + (`$":",FIXDIR,"/process.csv") 0: ( + "host,port,proctype,procname"; + "localhost,",string[PEERPORT-2],",selfproc,selfinst"; + "localhost,",string[PEERPORT],",otherproc,otherinst"; + "localhost,",string[DEADPORT],",deadproc,deadinst"); + }; + +teardownfixture:{[] killpeer[]; system "rm -rf ",FIXDIR;}; + +/ build the deps dict di.torq would assemble: injectables + this process's config slice. +svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;mocktimer;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; From efb35be5c94caf3b4ffbc6db21d2ce80cd0cc222 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Fri, 31 Jul 2026 16:11:56 +0100 Subject: [PATCH 03/10] aligned with di.toml module. Increased robustness of init guards, cut AI fluff code --- di/servers/servers.md | 6 ++++-- di/servers/servers.q | 19 ++++++++++--------- di/servers/test.csv | 4 +++- 3 files changed, 17 insertions(+), 12 deletions(-) diff --git a/di/servers/servers.md b/di/servers/servers.md index b85c3248..c4d893ce 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -47,7 +47,7 @@ h "1+1" | `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | -| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per export, for `di.torq` to register with `di.api`. | +| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta` plumbing omitted), for `di.torq` to register with `di.api`. | Export is deliberately conservative — only functions `di.torq` or a consumer actually calls (so `di.api` lists exactly these). The rest are **internal**: `retry` (the scheduled @@ -91,7 +91,9 @@ priority-ordered fan-out. - **`raiseerror` (log-then-signal)** for all post-init domain errors (`formathp` unknown ipctype, `selector` unknown selection, missing `process.csv`). `init`'s own dependency validation is the one exception (plain `'` — no logger yet). -- **`getapimeta`** exported; a test asserts it documents exactly the module's exports. No +- **`getapimeta`** exported; a test asserts it documents exactly the module's *callable* + exports — `init`/`getapimeta` are plumbing (di.torq calls them by convention) and are + deliberately omitted from the registry rows, matching di.toml and the skill convention. No `version` export / VERSION file yet — deferred to the di.depcheck rollout, as in di.config. - **Env-free** — di.servers reads no environment variable; the `process.csv` path arrives via `config`processcsv` (di.torq resolves it), holding di.config's env-free boundary. diff --git a/di/servers/servers.q b/di/servers/servers.q index 898b4fda..150f830c 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -59,6 +59,8 @@ init:{[deps] '"di.servers: handlers value must be a dict (see di.handlers)"]; if[not all `proctype`procname in key deps; '"di.servers: proctype and procname (self-identity) are required in deps"]; + if[not all -11h=type each deps`proctype`procname; + '"di.servers: proctype and procname must be symbols"]; .z.m.loginfo:deps[`log]`info; .z.m.logwarn:deps[`log]`warn; .z.m.logerr:deps[`log]`error; @@ -85,7 +87,7 @@ init:{[deps] formathp:{[host;port;ipctype] / internal - build a connection-handle symbol for `tcp`/`tcps`/`unix. only `tcp is exercised by / startup in v1; the others exist for a future SOCKETTYPE-style config. - h:string $[host=`localhost;`localhost;host]; + h:string host; p:string port; $[ipctype=`tcp; lower `$":",h,":",p; ipctype=`tcps;lower `$":tcps://",h,":",p; @@ -225,13 +227,12 @@ waitfortype:{[pt;timeoutms;pollms] }; getapimeta:{[] - / this module's api metadata, one row per exported function, for di.torq to register with di.api. - / names are bare (di.torq qualifies them). one (name;public;descrip;params;return) row per line. + / this module's api metadata, one row per CALLABLE API function, for di.torq to register with + / di.api. init/getapimeta are plumbing (di.torq calls them by convention) and are deliberately NOT + / listed - the registry describes the callable api, not plumbing. names are bare (di.torq qualifies). :flip `name`public`descrip`params`return!flip( - (`init; 0b; "wire injected deps + config and install the pc observer + retry job (idempotent)"; "[dict: deps - `log`timer`handlers + `proctype`procname (+`connections`processcsv)]"; "null"); - (`startup; 1b; "open connections to configured proctypes from process.csv (reads init config)"; "[]"; "null"); - (`getservers; 1b; "live SERVERS rows for a proctype"; "[symbol: proctype]"; "table: live server rows"); - (`gethandlebytype; 1b; "one live handle for a proctype via any/roundrobin/last selection"; "[symbol: proctype; symbol: selection]"; "int: handle, 0Ni if none"); - (`waitfortype; 1b; "block until a proctype connects or timeout elapses"; "[symbol: proctype; long: timeoutms; long: pollms]"; "boolean: 1b connected, 0b timed out"); - (`getapimeta; 0b; "this module's api metadata rows"; "[]"; "table: metadata rows")); + (`startup; 1b; "open connections to configured proctypes from process.csv (reads init config)"; "[]"; "null"); + (`getservers; 1b; "live SERVERS rows for a proctype"; "[symbol: proctype]"; "table: live server rows"); + (`gethandlebytype; 1b; "one live handle for a proctype via any/roundrobin/last selection"; "[symbol: proctype; symbol: selection]"; "int: handle, 0Ni if none"); + (`waitfortype; 1b; "block until a proctype connects or timeout elapses"; "[symbol: proctype; long: timeoutms; long: pollms]"; "boolean: 1b connected, 0b timed out")); }; diff --git a/di/servers/test.csv b/di/servers/test.csv index e769e7d9..652476e8 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -42,6 +42,8 @@ comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout -true,0,0,q,(asc key svc)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the exports +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;mocktimer;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype +true,0,0,q,(asc (key svc) except `init`getapimeta)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (plumbing omitted) +true,0,0,q,not any `init`getapimeta in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta not registered) true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapimeta rows carry the registry columns after,0,0,q,teardownfixture[],1,1,kill the peer and remove the fixture dir From eb7c1897c82ccbca4bbb611220290fa1bbf89fbe Mon Sep 17 00:00:00 2001 From: ascottDI Date: Wed, 5 Aug 2026 15:08:25 +0100 Subject: [PATCH 04/10] pulling in modules that are merged to main --- di/config/config.md | 265 ++++++++++++++++++++++++++++++++++++++++++ di/config/config.q | 177 ++++++++++++++++++++++++++++ di/config/init.q | 3 + di/config/test.csv | 81 +++++++++++++ di/dbwrite/dbwrite.md | 211 +++++++++++++++++++++++++++++++++ di/dbwrite/dbwrite.q | 236 +++++++++++++++++++++++++++++++++++++ di/dbwrite/init.q | 6 + di/dbwrite/test.csv | 246 +++++++++++++++++++++++++++++++++++++++ di/eodtime/eodtime.md | 157 +++++++++++++++++++++++++ di/eodtime/eodtime.q | 98 ++++++++++++++++ di/eodtime/init.q | 7 ++ di/eodtime/test.csv | 92 +++++++++++++++ di/log/init.q | 4 + di/log/log.md | 193 ++++++++++++++++++++++++++++++ di/log/log.q | 193 ++++++++++++++++++++++++++++++ di/log/test.csv | 218 ++++++++++++++++++++++++++++++++++ di/toml/init.q | 4 + di/toml/test.csv | 80 +++++++++++++ di/toml/test.q | 30 +++++ di/toml/toml.md | 104 +++++++++++++++++ di/toml/toml.q | 128 ++++++++++++++++++++ 21 files changed, 2533 insertions(+) create mode 100644 di/config/config.md create mode 100644 di/config/config.q create mode 100644 di/config/init.q create mode 100644 di/config/test.csv create mode 100644 di/dbwrite/dbwrite.md create mode 100644 di/dbwrite/dbwrite.q create mode 100644 di/dbwrite/init.q create mode 100644 di/dbwrite/test.csv create mode 100644 di/eodtime/eodtime.md create mode 100644 di/eodtime/eodtime.q create mode 100644 di/eodtime/init.q create mode 100644 di/eodtime/test.csv create mode 100644 di/log/init.q create mode 100644 di/log/log.md create mode 100644 di/log/log.q create mode 100644 di/log/test.csv create mode 100644 di/toml/init.q create mode 100644 di/toml/test.csv create mode 100644 di/toml/test.q create mode 100644 di/toml/toml.md create mode 100644 di/toml/toml.q diff --git a/di/config/config.md b/di/config/config.md new file mode 100644 index 00000000..f6a10549 --- /dev/null +++ b/di/config/config.md @@ -0,0 +1,265 @@ +# di.config + +Configuration loading and cascade resolution for the modular TorQ world. It replaces +TorQ's in-process config handling (`torq.q`: `loadf`, `loadconfig`, `loadaddconfig`, +`overrideconfig`). `di.config` resolves a settings **cascade** over a builtin root and an +app root — reading **both** flat `name:value` `.q` files and `.toml` files — and returns +the merged configuration as **one flat dict**. `di.torq` calls it once at startup +(`config:cascade[...]`) and hands that dict to each module's `init`. + +## The config model — one flat dict + +`di.config` resolves config into a **returned flat dict**; it does not execute settings +files into namespaces or hold any global store. This is the shape `di.torq` consumes: + +```q +config:use`di.config +cfg:config.cascade[builtinroot;approot;`rdb;`rdb1] / -> `subscribeto`rows`... !(...) +/ di.torq stamps identity and hands the whole dict to each module's init: +cfg:cfg,`proctype`procname!(`rdb;`rdb1) +rdb.init[cfg;`log`timer!(logdep;timerdep)] +``` + +`cascade` and `parsefile` are **pure** — no `init`, no logger, no globals — because +`di.torq` resolves config *before* the logger dependency has been built. Only +`overrideconfig` logs (it reports skipped/rejected overrides), so it — and only it — +requires `init`. + +## Precedence + +The cascade is resolved over **two roots** (`builtinroot`, `approot`) × a **three-tier +name sequence** (`default` → `{proctype}` → `{procname}`). Precedence is **name-major**: + +1. A more specific **name** always wins — `{procname}` beats `{proctype}` beats `default`. +2. **Within a name**, the later (app) root overrides the builtin root. +3. **Within a single tier** (one root + one name), the `.toml` file wins over the `.q` + file on a key clash — useful mid-migration when both formats coexist. + +So a key set in both `builtin/{proctype}` and `app/default` resolves to the **builtin +proctype** value, not the app default (name beats root). Effective load order, low → high +priority: + +``` +builtin/default(.q,.toml) → app/default → builtin/{proctype} → app/{proctype} + → builtin/{procname} → app/{procname} (each tier: .toml overrides .q) +``` + +`overrideconfig` sits on **top** of all of this — the command-line layer (see below). + +> **Note — name-major, not root-major.** The KDBX-POC's own vendored `di.config` used +> *root-major* precedence (all builtin tiers, then all app tiers), where `app/default` +> would beat `builtin/{proctype}`. `di.config` deliberately keeps the **name-major** rule +> TorQ has always used; `.toml` support was layered on without changing it. + +> **Note — avoid `-` in config file paths.** A source-level backtick symbol literal +> containing `-` parses as subtraction (`` `:/a-b.q `` → `` `:/a `` `- b.q`), not one +> symbol. The runtime string paths `cascade` builds internally are unaffected. + +## Settings file formats + +- **`.q`** — plain `name:value` lines. Split on the first `:`; the RHS is run through + `value`, so `` `:hdb ``, `` `trade`quote ``, `` `symbol$() `` all work. Blank lines and + `/`-comment lines are skipped. These are **flat** files (one process's config), **not** + the namespaced, executable settings files legacy TorQ `\l`-loaded. +- **`.toml`** — delegated to **`di.toml`**, loaded lazily (only when a `.toml` file is + actually present). TOML has no symbol type, so every TOML string comes back as a q + string, never a symbol — consumers that need a symbol normalise at the point of use + (`` `$ `` is a no-op on an already-a-symbol value). + +## Import and init + +`cascade` and `parsefile` need no `init` and no logger — call them directly: + +```q +config:use`di.config +cfg:config.cascade["/opt/torqx/di/torq/settings";"/opt/app/settings";`rdb;`rdb1] +``` + +`overrideconfig` logs, so wire a `log` dependency via `init` first — there is no silent +fallback. Pass an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` +loggers (context symbol, message string); the module performs **no** adaptation. + +```q +/ option 1: di.log — the standard logger (exports binary info/warn/error) +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) +config.init[enlist[`log]!enlist logdep] + +/ option 2: any hand-rolled binary {[c;m]} logger +mylog:`info`warn`error!( + {[c;m] -1 string[c],": INFO ",m;}; + {[c;m] -1 string[c],": WARN ",m;}; + {[c;m] -2 string[c],": ERROR ",m;}); +config.init[enlist[`log]!enlist mylog] +``` + +A logger that is *not* already binary `{[c;m]}` (e.g. a raw monadic +[`kx.log`](https://github.com/KxSystems/logging) instance) must be wrapped by the caller +first — di.config does no wrapping. + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `cascade` | `cascade[builtinroot;approot;proctype;procname]` | Resolve the settings cascade over the two roots × the `default`/`{proctype}`/`{procname}` name sequence and **return one flat dict**. Pure — no `init`, no logger, no globals. Precedence is name-major with `.toml > .q` within a tier (see above). A key set nowhere is simply absent; an all-missing cascade returns an empty dict. | +| `parsefile` | `parsefile[path]` | Parse a single settings file into a flat dict. A `.toml` path is delegated to `di.toml` (loaded lazily, behind the guard rail below); a `.q` path is flat `name:value` lines. A missing file (either extension) → empty dict. An existing `.toml` file with no `di.toml` on `QPATH` signals a clear error naming the file. Pure. | +| `overrideconfig` | `overrideconfig[config;params]` | Apply the **command-line override layer** on top of a resolved config dict — the top tier of the cascade. `di.torq` parses the process command line (`.Q.opt .z.x`) and calls this after `cascade`, so launch-time flags (e.g. `-loglevel info`, `-rows 5000`) win over the settings files. `config` is the dict from `cascade`; `params` is a dict keyed by setting name (symbol) with string (or list-of-string) values, each parsed into that setting's **existing** type. Only keys already present with a basic type are overridden; unknown keys, non-basic types, unparseable values, and multi-value overrides of a single-valued (scalar/string) setting are logged and skipped. A vector setting (e.g. a symbol list) accepts multiple values. Returns the **updated config dict**. Requires `init`. | +| `init` | `init[deps]` | Wire the injected logger for `overrideconfig`. `deps` is a dict with a required `log` key — an already-conforming binary `` `info`warn`error `` dict of `{[c;m]}` functions. Errors immediately if the log dependency is missing or malformed. | +| `getapimeta` | `getapimeta[]` | This module's API metadata — one `(name;public;descrip;params;return)` row per exported function, for `di.torq` to register with `di.api`. | + +Internal helpers — `parsetier` (per-tier `.q`+`.toml` merge), `applyoverride`, +`parsefailed`, and `hexchars` — are deliberately not exported. + +## Injectable dependencies + +| Injectable | Required keys | Function signature | +|---|---|---| +| `log` | `` `info`warn`error `` | `{[c;m]}` (context symbol, message string). Caller passes an already-conforming binary dict — built from `di.log` or hand-rolled. No adaptation is done in the module. Needed only by `overrideconfig`. | + +## Hard dependencies + +None mandatory. `cascade`/`parsefile` gain a **lazy, soft dependency on `di.toml`**: it is +`` use``d only when a `.toml` file is actually encountered, so the `.q`-only path needs +nothing. **di.toml does not need to be in the repo (or on `QPATH`) for di.config to load or +run** — it is required only at the moment a `.toml` settings file is actually parsed. + +### The `.toml` guard rail + +`parsefile` guards the di.toml load so a missing module produces a clear, actionable error +rather than a cryptic `notfound`. The order is deliberate: + +1. **Check the file exists first.** A missing file (either extension) → empty dict. So a + missing `.toml` tier *never* triggers the di.toml requirement — the internal + `parsetier`/`cascade` probe `base,".toml"` on every tier, and absent tiers cost nothing. +2. **Then, only for a `.toml` file that actually exists, require di.toml.** The internal + `requiretoml` helper attempts `` use`di.toml `` under protected evaluation; if it is not + found, it signals a clear error **naming the offending file**: + ``` + 'di.config: cannot parse TOML file '/path/to/x.toml' - the di.toml module was not found + on QPATH; a di.toml module is required to parse .toml settings (underlying: notfound: di.toml) + ``` +3. **Parse.** With di.toml resolved, the file is delegated to it. + +Because this signal happens at config-resolution time — *before* the logger dependency is +wired — it surfaces via the raised error (which aborts startup with the reason), not via the +injected logger. + +### Contract di.config expects from di.toml + +A di.toml module must export a **`parsefile`** function taking a settings-file path string +and returning a **flat dict** of `setting → value`: + +```q +(use`di.toml)[`parsefile] "/path/to/settings.toml" / -> `dir`rows!(":appdb";100) +``` + +TOML has no symbol type, so di.toml returns every TOML string as a **q char string** (never +a symbol); di.config applies no coercion, so consumers that need a symbol normalise at the +point of use (`` `$ `` is a no-op on an already-a-symbol value). + +> **⚠️ `di.toml` is not yet in kdbx-modules** — it currently lives only in the KDBX-POC and +> lands here with the PoC merge (a di.toml module is in development against the contract +> above). Until it is on `QPATH`, the `.toml` half of a tier is **dormant** in this repo: a +> `.toml` file that exists but cannot be parsed raises the guard error above; the `.q` path +> works standalone. Full `.toml` resolution is verified in the PoC (where `di.toml` is +> vendored); this repo's suite covers the guard's error path (di.toml absent) and the +> file-exists-before-delegation ordering. + +## Design notes & open gaps + +- **Returned dict, not a global store.** Resolution is a pure function returning a dict. + There is no queryable config store and no per-namespace `getmodule` — `di.torq` passes the + whole resolved dict to every module and each reads the keys it cares about. A missing key + is a caller bug (surfaced as a q null on an inline index), not something the config layer + silently papers over — every setting is expected to be booted with a default in the + `default` tier. +- **Retired: the executable-namespaced `.q` path.** An earlier revision also shipped a + `loadcascade`/`loadconfig`/`getmodule` path that `\l`-loaded namespaced, code-bearing `.q` + settings files (`.rdb.subscribeto:...`) into root namespaces and sliced them per module. + That parallel model was **removed** in favour of the single flat-dict cascade `di.torq` + actually consumes. If runtime config reload or an in-process store is ever needed, that is + a deliberate, separately-designed addition — not a silent revival of the globals path. +- **`overrideconfig` is the command-line layer, applied by `di.torq` — not a manual step.** + It mirrors TorQ's `overrideconfig`/`override[]`, which TorQ runs at startup off + `.Q.opt .z.x` so launch-time flags beat the settings files. di.config just does the + type-aware apply onto the resolved dict (which is why it stays env-free and never reads + `.z.x` itself). It deviates from TorQ deliberately on error handling: per-setting problems + (unknown name, non-basic type, value that won't parse to the target type) are logged and + skipped — a single bad override never aborts the batch and a null is never written. +- **`overrideconfig` parse-failure detection is type-aware.** Most basic types parse to a + null on a bad value, which `applyoverride` rejects via a null check. Boolean (`1h`) and + byte (`4h`) are the only in-scope basic types with **no null** — `"B"$"bad"` yields `0b`, + `"X"$"gg"` yields `0x00`, both non-null — so a bad value would slip past a null check and + silently corrupt config. `parsefailed` handles these two explicitly (boolean must be + `"0"`/`"1"`; byte must be even-length hex); anything else is rejected and logged, never + written. +- **String-typed (`10h`) settings are overridable — as text, no parse.** A char-string + setting's override value *is* already a string, so `applyoverride` takes it as-is rather + than casting it (any string is valid; there is no parse-failure case). This matters because + TOML has no symbol type, so a `.toml`-origin setting like `dir = ":appdb"` or + `loglevel = "info"` comes back as a string — without this, such settings could not be + command-line-overridden at all, whereas their `.q`-origin symbol equivalents (`` dir:`:appdb ``) + could. di.config stays policy-free: it preserves whatever type the setting already had + (string in, string out; symbol in, symbol out), leaving symbol-vs-string normalisation to + the consumer as everywhere else. +- **The sentinel merge key.** `cascade`'s accumulator is seeded with a sentinel + `` (enlist`)!enlist(::) `` key so its value list stays general from the first merge — a + same-typed value list coalesces to a typed vector that `,:` then refuses to widen (e.g. + adding a `` `symbol$() `` key to a dict whose values so far were all plain symbols). The + sentinel is dropped before the dict is returned. +- **Logging uses the three-flat-var convention** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, + called as `.z.m.loginfo[\`ctx;"msg"]` — matching `consistency.md` and `di.compression`. The + injected `log` value is still the same `` `info`warn`error `` dict; `init` fans it out into + the three module-local vars. + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.config +``` + +The `.toml` path is exercised in the KDBX-POC (where `di.toml` is vendored), not here — +see the hard-dependencies note above. + +## Follow-ups (deferred until other modules land) + +di.config is complete for v1 in isolation. The items below are intentionally deferred; +each is keyed to the module or milestone that unblocks it. + +### ⚠️ Logging call convention — NOT set in stone (ask before changing) + +di.config uses the **three-flat-var** convention (`.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`), +matching `consistency.md` and `di.compression`, but the project hasn't finalised this versus +the **single-dict** form (`.z.m.log[\`info][…]`). **Before changing di.config's logging — or +wiring logging in another module — stop and ask the user which convention is authoritative.** +Switching is mechanical (storage + call sites only; the injected input contract is identical). + +### When `di.toml` lands in kdbx-modules (with the PoC merge) + +- **Add committed `.toml` tests.** Today `di.toml` is not on this repo's `QPATH`, so a + committed `.toml`/mixed-tier test would fail here (verified in the PoC instead). Once + `di.toml` resolves, add tests covering: a `.toml` tier parsed into the dict, `.toml` + winning over `.q` within a tier, and a mixed `.q`+`.toml` cascade. + +### When `di.log` merges to main + +- **Add a committed `di.log` integration test.** The contract match (binary + `` `info`warn`error `` dict) is verified manually and stood in for by the kx.log-wrapper + emission test. Once `di.log` resolves on `QPATH`, add a test that builds the dict from + `di.log` and drives `overrideconfig` through it. + +### When `di.torq` is built + +- **Prove the end-to-end flow.** Add a multi-module integration test (under + `di.torq`/`di.inttest`, not here) exercising: `di.torq` resolves `KDBCONFIG`/`KDBAPPCONFIG` + → roots and `proctype`/`procname` → identity, calls `cascade`, then `overrideconfig` with + the command-line params, then hands the resolved dict to each module's `init`. +- **Hold the env-free boundary.** di.config reads no env vars or process identity — di.torq + owns that resolution and passes the roots and identity explicitly. + +### When `di.depcheck` is built (versioning rollout) + +- **Add a `version` export and `deps.q`.** When `di.depcheck` lands, add `version:"…"` to the + export and a minimal `deps.q` (di.config has no hard deps) as part of the coordinated + repo-wide rollout, not a di.config-only change. diff --git a/di/config/config.q b/di/config/config.q new file mode 100644 index 00000000..8feb594b --- /dev/null +++ b/di/config/config.q @@ -0,0 +1,177 @@ +/ configuration cascade resolution for the modular torq world - replaces torq.q's +/ loadf/loadconfig/loadaddconfig/overrideconfig. resolves a settings cascade over a builtin root +/ and an app root, reading both flat name:value .q files and .toml files, and RETURNS the merged +/ config as one flat dict (the shape di.torq consumes). reads no env/identity - the caller supplies +/ the roots and process identity. +/ precedence is name-major (more specific NAME wins; within a name the app root wins) and, within a +/ tier, .toml wins over .q. cascade/parsefile are pure (no init/logger) - di.torq resolves config +/ before the logger exists; only overrideconfig logs, so only it needs init. di.toml is a soft dep +/ (not yet in kdbx-modules): parsefile loads it lazily, only when a .toml file is present. + +init:{[deps] + / wire the injected logger (required by overrideconfig; no fallback). deps: a dict with a `log + / key holding a binary `info`warn`error dict of {[c;m]} loggers. cascade/parsefile don't need init. + if[99h<>type deps; + '"di.config: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.config: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.config: log value must be a dict; pass `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.config: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; + .z.m.loginfo[`init;"di.config initialised"]; + }; + +/ --- settings cascade resolution (.q + .toml -> flat dict) --- + +requiretoml:{[path] + / internal - resolve di.toml to parse a .toml file. di.toml is a soft dep (needed only once a + / .toml file appears); guard the load so a missing one gives a clear error naming the file, not a + / cryptic `notfound: di.toml`. returns the di.toml module. + :@[use;`di.toml;{[p;e] + msg:"di.config: cannot parse TOML file '",p,"' - the di.toml module was not found on QPATH; "; + msg,:"a di.toml module is required to parse .toml settings (underlying: ",e,")"; + 'msg + }[path;]]; + }; + +parsefile:{[path] + / parse ONE settings file into a flat dict. .toml -> di.toml (via requiretoml); .q -> flat + / `name:value` lines (split on first ":", RHS through value). blank, "/"-comment and non-pair + / (no ":") lines are skipped; a missing file -> empty dict. existence is checked FIRST, so a + / missing .toml tier never triggers the di.toml requirement. + fsym:`$":",path; + if[0=count key fsym; :()!()]; + if[path like "*.toml"; :(requiretoml[path])[`parsefile] path]; + lines:read0 fsym; + lines:lines where 0 empty. + (parsefile base,".q"),parsefile base,".toml" + }; + +cascade:{[builtinroot;approot;proctype;procname] + / resolve the cascade over the two roots x (default;proctype;procname) and RETURN the merged flat + / dict. pure (no init/logger/globals). name-major: a more specific NAME wins over the root layer, + / and within a name the app root wins (parsetier gives .toml>.q within a tier). the accumulator is + / seeded with a sentinel (`)!(::) key so its value list stays general from the first merge (a + / same-typed value list coalesces to a typed vector that ,: then won't widen); dropped before return. + / proctype/procname must be non-null symbol atoms: string`` is "", which would build a bogus + / "/" base and read a file named ".q"/".toml". a null means identity resolution failed, so + / surface it. (a null symbol is still type -11h, so the null check is separate from the type check.) + if[not all -11h=type each (proctype;procname); + '"di.config: cascade requires proctype and procname to be symbol atoms"]; + if[(null proctype)|null procname; + '"di.config: cascade requires non-null proctype and procname; got ",(-3!proctype)," and ",-3!procname]; + dirs:(builtinroot;approot); + / dedup (distinct keeps first-occurrence order, preserving least->most-specific). without it a + / repeated name re-applies its tier at a LATER slot: procname~proctype is redundant, but + / procname~`default would put default LAST and wrongly override proctype. + names:distinct `default,proctype,procname; + bases:raze {[ds;nm] ds,\:"/",string nm}[dirs;] each names; + acc:{[a;b] a,parsetier b}/[(enlist `)!enlist(::);bases]; + acc _ ` + }; + +/ --- command-line override layer (top of the cascade) --- + +hexchars:"0123456789abcdefABCDEF"; + +parsefailed:{[t;raw;vals] + / internal - true if any raw string failed to parse to type t. bool (1h) and byte (4h) have no + / null, so a bad parse ("B"$"bad"->0b, "X"$"gg"->0x00) slips past a null check - check their form + / explicitly. other types null on failure. + :$[1h=abs t;not all raw in (enlist"0";enlist"1"); + 4h=abs t;not all {(0=count[x] mod 2) and all x in hexchars} each raw; + any null vals]; + }; + +applyoverride:{[name;cur;raw] + / internal - parse raw into cur's type; return (applied;newvalue). cur's type drives the parse. + / applied is 0b (newvalue=cur) if cur is not a basic type, a value failed to parse, or a + / single-valued setting got other than one value. + t:type cur; + if[not (abs t) within (1;-1+count .Q.t); + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": not a basic type"]; + :(0b;cur); + ]; + raw:$[10h=type raw;enlist raw;raw]; + / scalar-atom (t<0) and string (10h) settings are single-valued: require exactly one value. reject + / a multi- or zero-value override rather than silently taking the first or writing a null. vector + / settings (t>0, not 10h) take as many as given. + if[(1<>count raw) and ((t<0)|(10h=t)); + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": expected a single value, got ",string count raw]; + :(0b;cur); + ]; + / a string (10h) is already text - take the override as-is (no parse). lets .toml-origin string + / settings be overridden like symbol (.q-origin) ones; type is preserved either way. + if[10h=t; + vals:first raw; + .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; + :(1b;vals); + ]; + vals:(upper .Q.t abs t)$'raw; + if[parsefailed[t;raw;vals]; + .z.m.logerr[`overrideconfig;"cannot override ",(string name),": value did not parse"]; + :(0b;cur); + ]; + / reduce to scalar only after parsefailed (which checks the whole list). + if[t<0;vals:first vals]; + .z.m.loginfo[`overrideconfig;"setting ",(string name)," to ",-3!vals]; + :(1b;vals); + }; + +overrideconfig:{[config;params] + / apply the command-line override layer onto a resolved config dict - the TOP cascade tier + / (di.torq calls this after cascade with .Q.opt .z.x, so launch flags win over files). config is + / the cascade dict; params keys settings (symbol) to string / string-list values, parsed into each + / setting's existing type. unknown keys, non-basic types and bad values are logged and skipped + / (never aborts the batch, never writes a null). returns the updated config dict. + if[99h<>type config; + .z.m.logerr[`overrideconfig;err:"di.config: config must be a dict"]; + 'err; + ]; + if[99h<>type params; + .z.m.logerr[`overrideconfig;err:"di.config: params must be a dict keyed by setting name"]; + 'err; + ]; + vars:key params; + if[0 merged flat dict (name-major, .toml>.q)"; "[string: builtinroot; string: approot; symbol: proctype; symbol: procname]"; "dict: merged setting -> value"); + (`overrideconfig; 1b; "apply the command-line override layer onto a resolved config dict"; "[dict: resolved config; dict: setting name (symbol) -> string or list of strings]"; "dict: updated config"); + (`parsefile; 1b; "parse one .q or .toml settings file into a flat dict"; "[string: file path (.q flat name:value, or .toml via di.toml)]"; "dict: setting -> value (empty if the file is missing)"); + (`getapimeta; 0b; "this module's api metadata rows"; "[]"; "table: metadata rows")); + }; diff --git a/di/config/init.q b/di/config/init.q new file mode 100644 index 00000000..2e06d014 --- /dev/null +++ b/di/config/init.q @@ -0,0 +1,3 @@ +/ configuration loading and cascade resolution for the modular torq world. +\l ::config.q +export:([init;cascade;overrideconfig;parsefile;getapimeta]) diff --git a/di/config/test.csv b/di/config/test.csv new file mode 100644 index 00000000..c257e8f7 --- /dev/null +++ b/di/config/test.csv @@ -0,0 +1,81 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,cfg:use`di.config,1,1,load the module +before,0,0,q,"logtab:([]lvl:`symbol$();ctx:`symbol$();msg:())",1,1,capturing log table +before,0,0,q,"mocklog:`info`warn`error!({[c;m]`logtab upsert(`info;c;m)};{[c;m]`logtab upsert(`warn;c;m)};{[c;m]`logtab upsert(`error;c;m)})",1,1,define a capturing binary logger +before,0,0,q,cfg.init[enlist[`log]!enlist mocklog],1,1,init with the required log dependency +before,0,0,q,"system""mkdir -p /tmp/diconfigbuiltin""",1,1,builtin root for the cascade +before,0,0,q,"system""mkdir -p /tmp/diconfigapp""",1,1,app root for the cascade +before,0,0,q,"(`:/tmp/diconfigbuiltin/default.q) 0: (""diconfiga:1"";""diconfigb:1"")",1,1,builtin/default flat name:value (two settings) +before,0,0,q,"(`:/tmp/diconfigbuiltin/rdb.q) 0: enlist ""diconfigb:10""",1,1,builtin/rdb proctype tier sets diconfigb +before,0,0,q,"(`:/tmp/diconfigapp/default.q) 0: (""diconfiga:100"";""diconfigb:5"")",1,1,app/default flat name:value (two settings) +before,0,0,q,"(`:/tmp/diconfigapp/rdb1.q) 0: enlist ""diconfiga:999""",1,1,app/rdb1 procname tier sets diconfiga +before,0,0,q,"basecfg:`enabled`rows`syms`tab`byte`str!(0b;100;`a`b;([]a:`long$());0x01;"":olddb"")",1,1,base config dict for the overrideconfig tests (str is a char-string setting, as a .toml value would be) +before,0,0,q,"system""mkdir -p /tmp/diconfigskip""",1,1,dir for the malformed-line skip test +before,0,0,q,"(`:/tmp/diconfigskip/default.q) 0: (""a:1"";"""";""/ a comment"";""straynocolon"";""b:2"")",1,1,a .q file mixing valid pairs with blank/comment/no-colon lines +before,0,0,q,"system""mkdir -p /tmp/diconfigtoml""",1,1,dir for the toml-guard test +before,0,0,q,"(`:/tmp/diconfigtoml/present.toml) 0: enlist ""a = 1""",1,1,a .toml file that exists but cannot be parsed without di.toml (absent in kdbx-modules) +comment,,,,,,,init - dependency validation +fail,0,0,q,cfg.init[(::)],1,1,init rejects a non-dictionary deps +fail,0,0,q,cfg.init[enlist[`foo]!enlist 1],1,1,init rejects deps without a log key +fail,0,0,q,cfg.init[enlist[`log]!enlist 42],1,1,init rejects a non-dict log value +fail,0,0,q,cfg.init[enlist[`log]!enlist ((enlist`info)!enlist {[c;m]})],1,1,init rejects a log dict missing warn/error +comment,,,,,,,parsefile - parse one flat .q settings file into a dict (.toml delegated to di.toml) +true,0,0,q,"1~cfg.parsefile[""/tmp/diconfigbuiltin/default.q""]`diconfiga",1,1,parsefile reads a flat name:value .q file into a dict +true,0,0,q,"2=count cfg.parsefile[""/tmp/diconfigbuiltin/default.q""]",1,1,parsefile returns every setting in the file +true,0,0,q,"0=count cfg.parsefile[""/tmp/diconfignope/default.q""]",1,1,parsefile returns an empty dict for a missing .q file +true,0,0,q,"(`a`b!1 2)~cfg.parsefile[""/tmp/diconfigskip/default.q""]",1,1,parsefile skips blank/comment/no-colon lines and returns only valid pairs (no-colon line must not crash the split) +comment,,,,,,,parsefile - .toml guard rail (di.toml is a soft dep; absent in kdbx-modules so the error path is exercised here) +true,0,0,q,"0=count cfg.parsefile[""/tmp/diconfignope/absent.toml""]",1,1,a MISSING .toml contributes nothing and does NOT require di.toml (existence checked before delegation) +fail,0,0,q,"cfg.parsefile[""/tmp/diconfigtoml/present.toml""]",1,1,an EXISTING .toml with no di.toml on QPATH signals rather than parsing +true,0,0,q,"(@[cfg.parsefile;""/tmp/diconfigtoml/present.toml"";{x}]) like ""*di.toml module was not found*""",1,1,the .toml guard error names the missing di.toml module +comment,,,,,,,cascade - merged flat dict over two roots x default/proctype/procname (name-major; .toml>.q within a tier) +true,0,0,q,"100~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfiga",1,1,within a name the later (app) root overrides the builtin root +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdbnofile])`diconfigb",1,1,name-major: builtin proctype beats app default (would be 5 under root-major) +true,0,0,q,"999~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1])`diconfiga",1,1,name-major: the procname tier (app/rdb1) wins over every less-specific name +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1])`diconfigb",1,1,a key absent from the procname tier keeps its most-specific set value (builtin proctype) +true,0,0,q,"not (`) in key cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb1]",1,1,the sentinel merge key is dropped from the returned dict +true,0,0,q,"0=count cfg.cascade[""/tmp/diconfignope1"";""/tmp/diconfignope2"";`rdb;`rdb1]",1,1,an all-missing cascade resolves to an empty dict +true,0,0,q,"10~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`default])`diconfigb",1,1,dedup: procname=`default must NOT re-apply the default tier last and override proctype (would be 5 without distinct) +true,0,0,q,"100~(cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;`rdb])`diconfiga",1,1,dedup: proctype~procname resolves correctly (each tier once) +fail,0,0,q,"cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";`rdb;(`)]",1,1,cascade rejects a null procname (else it resolves a bogus root/.q tier) +fail,0,0,q,"cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";(`);`rdb1]",1,1,cascade rejects a null proctype +fail,0,0,q,"cfg.cascade[""/tmp/diconfigbuiltin"";""/tmp/diconfigapp"";""rdb"";`rdb1]",1,1,cascade rejects a non-symbol (string) proctype +comment,,,,,,,overrideconfig - command-line override layer applied onto a resolved config dict +true,0,0,q,"1b~(cfg.overrideconfig[basecfg;`enabled`rows!(enlist""1"";enlist""500"")])`enabled",1,1,bool parsed and set from string +true,0,0,q,"500~(cfg.overrideconfig[basecfg;`enabled`rows!(enlist""1"";enlist""500"")])`rows",1,1,long parsed and set from string +true,0,0,q,"basecfg~cfg.overrideconfig[basecfg;()!()]",1,1,empty params returns the config unchanged +true,0,0,q,"0b~(cfg.overrideconfig[basecfg;enlist[`enabled]!enlist ""bad""])`enabled",1,1,unparseable boolean is rejected - the value is left unchanged (not silently set to 0b via a null) +true,0,0,q,"any (exec msg from logtab) like ""*did not parse*""",1,1,rejected boolean override is logged +true,0,0,q,"0xff~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""ff""])`byte",1,1,valid hex byte parsed and applied +true,0,0,q,"0x01~(cfg.overrideconfig[basecfg;enlist[`byte]!enlist ""gg""])`byte",1,1,invalid hex byte is rejected (no null - must not slip through as 0x00) +true,0,0,q,"`xx`yy~(cfg.overrideconfig[basecfg;enlist[`syms]!enlist (""xx"";""yy"")])`syms",1,1,symbol list parsed and set from strings +true,0,0,q,""":newdb""~(cfg.overrideconfig[basecfg;enlist[`str]!enlist "":newdb""])`str",1,1,a string-typed setting is overridden as-is (this is what makes .toml string settings overridable) +true,0,0,q,"10h~type (cfg.overrideconfig[basecfg;enlist[`str]!enlist "":newdb""])`str",1,1,string override preserves the string (10h) type +true,0,0,q,"100~(cfg.overrideconfig[basecfg;enlist[`rows]!enlist (""55"";""66"")])`rows",1,1,a multi-value override on a scalar setting is rejected (not silently first) - left unchanged +true,0,0,q,""":olddb""~(cfg.overrideconfig[basecfg;enlist[`str]!enlist (""aa"";""bb"")])`str",1,1,a multi-value override on a string setting is rejected - left unchanged +true,0,0,q,"any (exec msg from logtab) like ""*expected a single value*""",1,1,a multi-value override on a single-valued setting is logged +true,0,0,q,"not `nope in key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting is not added to the config dict +true,0,0,q,"(key basecfg)~key cfg.overrideconfig[basecfg;enlist[`nope]!enlist enlist""1""]",1,1,an unknown setting leaves the config keys unchanged +true,0,0,q,"any (exec msg from logtab) like ""*unknown setting*""",1,1,unknown setting logged +true,0,0,q,"(cfg.overrideconfig[basecfg;enlist[`tab]!enlist enlist""1""])[`tab]~basecfg`tab",1,1,a non-basic-type setting (table) is left unchanged +true,0,0,q,"any (exec msg from logtab) like ""*not a basic type*""",1,1,non-basic-type setting logged +fail,0,0,q,cfg.overrideconfig[42;()!()],1,1,overrideconfig rejects a non-dict config +fail,0,0,q,cfg.overrideconfig[basecfg;42],1,1,overrideconfig rejects non-dict params +fail,0,0,q,"cfg.overrideconfig[basecfg;enlist[1]!enlist enlist""x""]",1,1,overrideconfig rejects non-symbol params keys +comment,,,,,,,getapimeta - module api metadata +true,0,0,q,(asc key cfg)~asc exec name from cfg.getapimeta[],1,1,getapimeta documents exactly the module's exports +true,0,0,q,`name`public`descrip`params`return~cols cfg.getapimeta[],1,1,getapimeta rows carry the registry columns +true,0,0,q,1b~(1!cfg.getapimeta[])[`cascade]`public,1,1,a public API function (cascade) is marked public +true,0,0,q,0b~(1!cfg.getapimeta[])[`init]`public,1,1,framework plumbing (init) is not public +comment,,,,,,,logger emission - real logger via a caller-side binary wrapper (must run last; re-inits cfg) +run,0,0,q,kxlogger:use`kx.log,1,1,load the kx.log module +run,0,0,q,kxinst:kxlogger.createLog[],1,1,create a real kx.log instance +run,0,0,q,kxh:hopen `:/tmp/diconfigkxsink.txt,1,1,open a file sink to capture output +run,0,0,q,kxinst.add[kxh;`info],1,1,route info-level output to the file sink +run,0,0,q,"kxbinary:`info`warn`error!({[c;m]kxinst[`info][string[c],"": "",m]};{[c;m]kxinst[`warn][string[c],"": "",m]};{[c;m]kxinst[`error][string[c],"": "",m]})",1,1,wrap the monadic logger into the binary {[c;m]} contract +run,0,0,q,cfg.init[enlist[`log]!enlist kxbinary],1,1,re-init di.config with the conforming binary logger +run,0,0,q,"kxres:cfg.overrideconfig[enlist[`kxrows]!enlist 5;enlist[`kxrows]!enlist enlist""7""]",1,1,drive a real info-level log through the wrapped logger +run,0,0,q,hclose kxh,1,1,flush and close the file sink +true,0,0,q,7~kxres`kxrows,1,1,the override was applied (returned dict carries the new value) +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*overrideconfig: setting*""",1,1,the override emitted an info line through the real logger +true,0,0,q,"any (read0 `:/tmp/diconfigkxsink.txt) like ""*kxrows*""",1,1,emitted message names the overridden setting diff --git a/di/dbwrite/dbwrite.md b/di/dbwrite/dbwrite.md new file mode 100644 index 00000000..b5af7fda --- /dev/null +++ b/di/dbwrite/dbwrite.md @@ -0,0 +1,211 @@ +# di.dbwrite + +Write, sort, and attribute utilities for kdb+ processes that persist data to disk (rdb, wdb, tickerlogreplay). + +A **config table** (`tabname`,`att`,`column`,`sort`) drives which columns are sorted and which attributes are applied per table. Load it once with `readcsv` (from a CSV) or `setconfig` (from an in-memory table); `sort` and `savedown` read from module state automatically. A table with no explicit entry falls back to a `default` row; if no config has been loaded, the built-in default (sort every table by `time` ascending) applies. + +--- + +## Features + +- Write an in-memory table to a date-partitioned HDB with `savedown` — enumerates syms, writes, sorts per stored config, then runs `.Q.gc[]` +- Append rows to an existing partition with `appenddown` — enumerates syms and appends; sort separately when the partition is complete +- Sort on-disk table partitions by configured columns using `xasc`, then apply kdb+ attributes (`p`,`s`,`g`,`u`) +- Config loaded once via `readcsv` (from a CSV) or `setconfig` (from a table); inspectable at any time with `getconfig` +- Sort and attribute errors are caught-and-logged (a single partition failure does not halt the run); config and write errors are raised to the caller with a `di.dbwrite:` prefix + +--- + +## Dependencies + +| Dependency | Key | Required | Description | +|---|---|---|---| +| logger | `` `log `` | yes | Functions `info`,`warn`,`error` — each binary `{[c;m] ...}` (context symbol, message string) | + +The `log` dependency must be passed to `init`. The module throws if it is absent, `(::)`, or missing any of the three keys. The `log` value must **already** be a binary `` `info`warn`error!{[c;m]} `` dict — each function takes a context symbol `c` (the calling function) and a message string `m`. `init` performs **no** adaptation and fans the dict out into `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`. Build it from `di.log` (which exports binary `info`/`warn`/`error`) or hand-roll one; a raw monadic [`kx.log`](https://github.com/KxSystems/logging) instance must be wrapped by the caller first — the module will not do it. + +```q +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) +dbwrite:use`di.dbwrite +dbwrite.init[enlist[`log]!enlist logdep] + +/ a raw kx.log instance is monadic - wrap it to binary {[c;m]} before passing: +/ loginst:(use`kx.log)[`createLog][] +/ `info`warn`error!({[c;m]loginst[`info][string[c],": ",m]};…) +``` + +--- + +## The config table + +| Column | Type | Description | +|---|---|---| +| `tabname` | symbol | Table name, or `` `default `` as a catch-all fallback | +| `att` | symbol | Attribute applied after sort: `p`,`s`,`g`,`u`, or empty (`` ` ``) for none | +| `column` | symbol | Column to sort and/or attribute | +| `sort` | boolean | `1b` — include in the `xasc` sort key; `0b` — attribute only | + +Build it directly and load with `setconfig`, or read it from a CSV with `readcsv`. A CSV must have exactly the four columns `tabname,att,column,sort` in **any** order (the result is normalised to canonical order); a missing, extra, or misnamed column raises a clear `di.dbwrite:` error rather than silently mis-parsing. + +``` +tabname,att,column,sort +trade,p,sym,1 +trade,,price,0 +default,,time,1 +``` + +--- + +## Functions + +| Function | Description | +|---|---| +| `init[deps]` | Wire injected dependencies; must be called first | +| `readcsv[file]` | Read a config CSV and store it in module state | +| `setconfig[t]` | Store a hand-built config table in module state | +| `getconfig[]` | Return the currently stored config (`::` if not yet set) | +| `sort[tabname;dirs]` | Sort on-disk partition(s) for a table and apply attributes per stored config | +| `savedown[dir;part;tabname;data]` | Write an in-memory table to an HDB partition, sort it, then run gc | +| `appenddown[dir;part;tabname;data]` | Append rows to an existing partition (no sort) | +| `applyattr[dloc;colname;att]` | Apply a single kdb+ attribute to an on-disk column | + +--- + +### `init[deps]` + +Wire injected dependencies. Must be called before any other function. Also resets the stored sort config to `(::)`. + +| Arg | Type | Description | +|---|---|---| +| `deps` | dict | Must contain `` `log `` → `` `info`warn`error!(infofn;warnfn;errfn) `` | + +Throws (prefixed `di.dbwrite:`) if `deps` is not a dict, `log` is missing, or the log dict lacks any required key. + +--- + +### `readcsv[file]` + +Read a config CSV and store it in module state (equivalent to calling `setconfig` with the parsed result). The stored config is used by subsequent calls to `sort` and `savedown`. + +| Parameter | Type | Description | +|---|---|---| +| `file` | symbol/hsym/string | Path to the CSV. Symbols are coerced with `hsym`; strings are converted via `hsym `$`. Errors (`di.dbwrite:`) if not a symbol or string. | + +The CSV is loaded with `0:` — the header row names the columns, so their order does not matter — and the resulting table is validated by `checkconfig` before storing. A clear `di.dbwrite:` error is raised if the columns are not exactly `tabname`,`att`,`column`,`sort`, any `tabname`/`column` is null, `sort` is not boolean, or any `att` value is not in `` ` `p`s`g`u ``. Config CSVs are expected to be well-formed: `0:` silently pads/truncates ragged rows and coerces non-`0`/`1` sort values, so those are not separately rejected. + +```q +dbwrite.readcsv `:config/sort.csv +dbwrite.readcsv "config/sort.csv" +``` + +--- + +### `setconfig[t]` + +Store a hand-built config table in module state. Alternative to `readcsv` when the config is constructed in-session rather than read from a file. Validates the table before storing — throws (`di.dbwrite:`) on any schema or content error. + +| Parameter | Type | Description | +|---|---|---| +| `t` | table | Config table with columns `tabname`,`att`,`column`,`sort` | + +```q +dbwrite.setconfig ([] tabname:`trade`default; att:`p`; column:`sym`time; sort:11b) +``` + +--- + +### `getconfig[]` + +Return the currently stored sort config. Returns `(::)` if `init` has been called but neither `readcsv` nor `setconfig` has been called yet. + +```q +dbwrite.getconfig[] +``` + +--- + +### `sort[tabname;dirs]` + +Sort and apply attributes to the on-disk partition(s) for one table, using the config stored by `readcsv` or `setconfig`. Falls back to the built-in default (sort by `time` ascending) if no config has been loaded. + +| Parameter | Type | Description | +|---|---|---| +| `tabname` | symbol | Table name. Errors (`di.dbwrite:`) if not a symbol. | +| `dirs` | hsym, or list of hsyms | Partition directory or directories. | + +Row lookup: the table's own rows → the `default` row → otherwise a warn is logged and `()` returned. Each partition is processed independently; a failure on one is logged and does not halt the rest. + +```q +dbwrite.sort[`trade; (`:hdb/2024.01.02/trade; `:hdb/2024.01.03/trade)] +``` + +--- + +### `savedown[dir;part;tabname;data]` + +Write an in-memory table to a date-partitioned HDB partition, sort it per the stored config, then run `.Q.gc[]`. Enumerates symbol columns against the HDB sym file before writing. Sorting and attributes are applied by `sort` *after* the write — so an attribute like `p#` is only applied once its column is correctly grouped. + +| Parameter | Type | Description | +|---|---|---| +| `dir` | hsym | HDB root directory (e.g. `` `:hdb ``). | +| `part` | date/month/int | Partition value. | +| `tabname` | symbol | Table name — determines the partition subdirectory. | +| `data` | table | In-memory table to write. | + +Throws on write failure. + +```q +dbwrite.savedown[`:hdb; 2024.01.02; `trade; data] +``` + +--- + +### `appenddown[dir;part;tabname;data]` + +Append rows to an existing on-disk partition (enumerates syms); does **not** sort. Keeping sort separate allows multiple intraday appends without re-sorting a growing partition on each call — sort once when the partition is complete. + +| Parameter | Type | Description | +|---|---|---| +| `dir` | hsym | HDB root directory. | +| `part` | date/month/int | Partition value. | +| `tabname` | symbol | Table name. | +| `data` | table | Rows to append. | + +Throws (prefixed `di.dbwrite:`) if the partition does not exist, or on write failure. + +```q +/ intraday: append each batch as it arrives +dbwrite.appenddown[`:hdb; 2024.01.02; `trade; batch] +/ end-of-day: sort once when done +dbwrite.sort[`trade; .Q.par[`:hdb; 2024.01.02; `trade]] +``` + +--- + +### `applyattr[dloc;colname;att]` + +Apply a single kdb+ attribute to one on-disk column (best-effort: logs and swallows errors so a run continues). A non-attribute `att` (the empty sentinel or any value outside `` `p`s`g`u ``) is a silent no-op. + +```q +dbwrite.applyattr[`:hdb/2024.01.02/trade; `sym; `p] +``` + +--- + +## Running tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.dbwrite +``` + +The suite injects binary mock loggers (`{[c;m] ...}`): a no-op logger, and a capturing logger that records `(level;context;msg)` so log behaviour can be asserted. It also wires a real `kx.log` instance (wrapped caller-side into the binary `{[c;m]}` contract) through `init` to confirm the module works end-to-end against the system logger. On-disk behaviour (sort, attributes, `savedown`/`appenddown`) is exercised against real splayed partitions and cleaned up afterwards. It covers: dependency-injection validation; `readcsv` stores-and-verifies / column-order independence / header-validation failures; `setconfig` happy path and validation failures; `sort` edge cases / resolution / on-disk results / multi-dir / non-fatal partition failure; `savedown` write+sort (default and explicit config, and a table without `sym`); `appenddown` append-without-sort then explicit sort, and the non-existent-partition error; `applyattr`; the binary `{[c;m]}` context-tagged logging contract; and the real `kx.log` integration. + +--- + +## Exported symbols + +```q +export:([init;readcsv;setconfig;getconfig;sort;applyattr;savedown;appenddown]) +``` diff --git a/di/dbwrite/dbwrite.q b/di/dbwrite/dbwrite.q new file mode 100644 index 00000000..3eb6306c --- /dev/null +++ b/di/dbwrite/dbwrite.q @@ -0,0 +1,236 @@ +/ dbwrite - write, sort, and attribute utilities for on-disk data +/ used by processes that persist data to disk (rdb, wdb, tickerlogreplay) + +/ attributes that may legitimately appear in a config (empty leaves a column unattributed) +validatts:``p`s`g`u; + +/ built-in fallback config - sort every table by time ascending when no config is supplied +defaultparams:([] tabname:enlist`default; att:enlist`; column:enlist`time; sort:enlist 1b); + +init:{[deps] + / wire the injected logger - required, no silent fallback. deps: a dict with a `log key + / holding a binary `info`warn`error dict of {[c;m]} loggers (context symbol, message string), + / from di.log or hand-rolled. no adaptation here, so a monadic kx.log instance must be wrapped + / first. e.g. di.dbwrite.init[enlist[`log]!enlist logdep] + if[99h<>type deps; + '"di.dbwrite: deps must be a dict with a `log key"]; + if[not `log in key deps; + '"di.dbwrite: log dependency is required; pass `info`warn`error functions keyed on `log"]; + if[99h<>type deps`log; + '"di.dbwrite: log value must be a dict of `info`warn`error functions"]; + if[not all (`info`warn`error) in key deps`log; + '"di.dbwrite: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; + .z.m.loginfo:deps[`log]`info; + .z.m.logwarn:deps[`log]`warn; + .z.m.logerr:deps[`log]`error; + .z.m.sortconfig:(::); + }; + +readcsv:{[file] + / read a sort-config csv and store it in .z.m.sortconfig; used by sort and savedown + / the csv must have the columns tabname,att,column,sort (in any order) + / file: hsym, bare symbol, or string path + if[10h=type file; file:hsym `$file]; + if[-11h=type file; if[not ":" = first string file; file:hsym file]]; + if[not -11h=type file; + .z.m.logerr[`readcsv;err:"di.dbwrite: readcsv file must be a symbol or string path, got type ",string type file]; + 'err; + ]; + t:parsecsv @[readfile; file; readerr[file]]; + checkconfig t; + .z.m.loginfo[`readcsv;"read ",(string count t)," sort config row(s) from ",string file]; + .z.m.sortconfig:t; + }; + +setconfig:{[t] + / set .z.m.sortconfig from an in-memory table; alternative to readcsv when config is built in-session + / t must be a table with columns tabname,att,column,sort + checkconfig t; + .z.m.sortconfig:t; + }; + +getconfig:{[] + / return the current sort config stored in .z.m.sortconfig; (::) if not yet set + :.z.m.sortconfig; + }; + +/ internal - protected file read; only the i/o so a genuine read failure gets the readerr message +readfile:{[file] + / returns the raw csv lines; header validation and parsing happen in parsecsv + .z.m.loginfo[`readfile;"reading sort config from ",string file]; + :read0 file; + }; + +/ internal - log and rethrow a csv read failure +readerr:{[file;e] + / build the message once, log it under the read context, then rethrow it to the caller + m:"di.dbwrite: failed to read ",string[file],": ",e; + .z.m.logerr[`readerr;m]; + 'm; + }; + +/ internal - parse csv lines into a config table +parsecsv:{[lines] + / load via 0: - the header row (enlist delim) names the columns, so column order does not + / matter; cast sort to boolean if present. structural validation is left to checkconfig. + t:((count "," vs first lines)#"S";enlist",") 0: lines; + :$[`sort in cols t; update sort:"B"$string sort from t; t]; + }; + +/ internal - validate a sort-config table, signalling a clear error if it is malformed +checkconfig:{[t] + / guards every sort call so a hand-built or csv-derived table is rejected early if wrong + if[98h<>type t; + .z.m.logerr[`checkconfig;err:"di.dbwrite: config must be a table with columns `tabname`att`column`sort"]; + 'err; + ]; + c:cols t; + badcols:c where not c in `tabname`att`column`sort; + if[count badcols; + .z.m.logerr[`checkconfig;err:"di.dbwrite: unrecognised config column(s): ",", " sv string badcols]; + 'err; + ]; + missingcols:(`tabname`att`column`sort) where not (`tabname`att`column`sort) in c; + if[count missingcols; + .z.m.logerr[`checkconfig;err:"di.dbwrite: missing required config column(s): ",", " sv string missingcols]; + 'err; + ]; + if[any null t`tabname; + .z.m.logerr[`checkconfig;err:"di.dbwrite: config tabname must not be null"]; + 'err; + ]; + if[any null t`column; + .z.m.logerr[`checkconfig;err:"di.dbwrite: config column must not be null"]; + 'err; + ]; + if[not 1h=type t`sort; + .z.m.logerr[`checkconfig;err:"di.dbwrite: the sort column must be boolean"]; + 'err; + ]; + badatts:at where not (at:distinct t`att) in validatts; + if[count badatts; + .z.m.logerr[`checkconfig;err:"di.dbwrite: unrecognised attribute(s) in att column: ",", " sv string badatts]; + 'err; + ]; + }; + +sort:{[tabname;dirs] + / sort and apply attributes to the on-disk partition dirs for one table using .z.m.sortconfig + / both the sort column order and the att assignments (p/s/g/u) are driven by the config + / falls back to defaultparams if config has not been set via readcsv or setconfig + / tabname: symbol; dirs: hsym or list of hsyms (partition directories e.g. from .Q.par) + config:$[(::)~.z.m.sortconfig; defaultparams; .z.m.sortconfig]; + checkconfig config; + if[not -11h=type tabname; + .z.m.logerr[`sort;err:"di.dbwrite: tabname must be a symbol, got type ",string type tabname]; + 'err; + ]; + st:string tabname; + .z.m.loginfo[`sort;"sorting the ",st," table"]; + sp:getsortparams[config;tabname]; + if[not count sp; :()]; + sortdir[sp] each distinct (),dirs; + .z.m.loginfo[`sort;"finished sorting the ",st," table"]; + }; + +/ internal - resolve which config rows apply to a table +getsortparams:{[config;tab] + / tab is the table-name symbol (NOT a table); named to avoid clashing with the tabname column + / a table uses its own rows; unlisted tables fall back to the default row, else are skipped + if[count tabsp:select from config where tabname=tab; + .z.m.loginfo[`getsortparams;"sort params found for: ",string[tab]]; + :tabsp; + ]; + if[count defsp:select from config where tabname=`default; + .z.m.loginfo[`getsortparams;"no sort params for: ",string[tab],"; using defaults"]; + :defsp; + ]; + .z.m.logwarn[`getsortparams;"no sort params for: ",string[tab],"; skipping sort"]; + :0#config; + }; + +/ internal - log a sort failure without rethrowing so remaining partitions still run +sorterr:{[sc;dl;e] + / a single partition failure should not halt the whole run + .z.m.logerr[`sorterr;"failed to sort ",string[dl]," by ",(", " sv string sc),": ",e]; + :(); + }; + +/ internal - sort one partition directory by the given columns +sortcolumns:{[dloc;sortcols] + / split out of sortdir so the conditional body there stays a single statement + .z.m.loginfo[`sortcolumns;"sorting ",string[dloc]," by: ",", " sv string sortcols]; + .[xasc;(sortcols;dloc); + sorterr[sortcols;dloc]]; + }; + +/ internal - sort columns and apply attributes for a single on-disk partition directory +sortdir:{[sp;dloc] + / sort by the columns flagged sort=1b, then hand every row to applyattr (it skips non-attributes) + sortcols:exec column from sp where sort, not null column; + if[count sortcols; sortcolumns[dloc;sortcols]]; + applyattr[dloc;;]'[sp`column;sp`att]; + }; + +/ internal - log an attribute application failure without rethrowing +attrerr:{[dl;cn;at;e] + / logs failure and continues so other columns and partitions still get processed + .z.m.logerr[`attrerr;"unable to apply ",string[at]," attr to ",string[cn]," in ",string[dl],": ",e]; + :(); + }; + +applyattr:{[dloc;colname;att] + / apply a single kdb+ attribute to an on-disk column; logs and swallows errors so a run continues + / dloc: hsym (partition directory e.g. `:hdb/2024.01.01/trade); colname: symbol; att: symbol (p|s|g|u or empty) + / skip anything that is not a real attribute - covers the empty none-sentinel and any bad value + if[not att in `p`s`g`u; :()]; + .z.m.loginfo[`applyattr;"applying ",string[att]," attr to ",string[colname]," in ",string dloc]; + .[{@[x;y;z#]}; + (dloc;colname;att); + attrerr[dloc;colname;att]]; + }; + +savedown:{[dir;part;tabname;data] + / write an in-memory table to a date-partitioned hdb partition, then sort it per .z.m.sortconfig + / dir: hdb root (hsym); part: partition value (date/month/int); tabname: symbol; data: in-memory table + / enumerates syms against the hdb sym file; sorting and attributes are driven by .z.m.sortconfig + .z.m.loginfo[`savedown;"saving ",string[tabname]," partition ",string[part]," to ",string dir]; + path:` sv (.Q.par[dir;part;tabname];`); + path set .Q.en[dir;data]; + sort[tabname;path]; + .z.m.loginfo[`savedown;"finished saving ",string tabname]; + gc[]; + }; + +appenddown:{[dir;part;tabname;data] + / append rows to an existing on-disk partition (enumerates syms); does not sort + / call sort separately once the partition is complete, to avoid re-sorting on every append + / dir: hdb root (hsym); part: partition value; tabname: symbol; data: in-memory table + .z.m.loginfo[`appenddown;"appending ",string[tabname]," partition ",string[part]," in ",string dir]; + path:` sv (.Q.par[dir;part;tabname];`); + if[not count @[key;path;{`$()}]; + .z.m.logerr[`appenddown;err:"di.dbwrite: appenddown partition does not exist at ",string path]; + 'err; + ]; + .[path;();,;.Q.en[dir;data]]; + .z.m.loginfo[`appenddown;"finished appending ",string tabname]; + }; + +/ internal - render a memory-usage dict as a "key=val MB; ..." string +fmtmem:{[m] + / m: dict of MB values keyed by .Q.w field name + :"; " sv "=" sv' flip (string key m; (string value m),\:" MB"); + }; + +/ format current process memory stats as a loggable string +memstats:{[] + / convert .Q.w[] (bytes) to MB and render it via fmtmem + :"mem stats: ",fmtmem `long$.Q.w[]%1048576; + }; + +gc:{[] + / run .Q.gc[] and log before/after memory stats + .z.m.loginfo[`gc;"starting garbage collect. ",memstats[]]; + r:.Q.gc[]; + .z.m.loginfo[`gc;"garbage collection returned ",(string `long$r%1048576),"MB. ",memstats[]]; + }; \ No newline at end of file diff --git a/di/dbwrite/init.q b/di/dbwrite/init.q new file mode 100644 index 00000000..6bf98f85 --- /dev/null +++ b/di/dbwrite/init.q @@ -0,0 +1,6 @@ +/ dbwrite module - write, sort, and attribute utilities for on-disk data +/ used by processes that persist data to disk (rdb, wdb, tickerlogreplay) + +\l ::dbwrite.q + +export:([init;readcsv;setconfig;getconfig;sort;applyattr;savedown;appenddown]) diff --git a/di/dbwrite/test.csv b/di/dbwrite/test.csv new file mode 100644 index 00000000..f6df53fa --- /dev/null +++ b/di/dbwrite/test.csv @@ -0,0 +1,246 @@ +action,ms,bytes,lang,code,repeat,minver,comment +before,0,0,q,dbwrite:use`di.dbwrite,1,1,load module +before,0,0,q,mylog:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,define no-op binary mock logger {[c;m]} +before,0,0,q,"caplog:([] fn:`symbol$();ctx:`symbol$();msg:())",1,1,initialise log capture table +before,0,0,q,logcap:`info`warn`error!({[c;m] `caplog upsert(`info;c;m)};{[c;m] `caplog upsert(`warn;c;m)};{[c;m] `caplog upsert(`error;c;m)}),1,1,define capturing binary mock logger {[c;m]} +before,0,0,q,"rmrf:{[d] if[11h=type k:key d; rmrf each ` sv/:d,/:k]; @[hdel;d;{}]}",1,1,recursive delete helper for on-disk cleanup +before,0,0,q,"`:tmp_dbw_valid.csv 0: (""tabname,att,column,sort"";""trade,p,sym,1"";""trade,,time,0"";""default,p,sym,1"")",1,1,write valid sort csv +before,0,0,q,"`:tmp_dbw_badcols.csv 0: (""tabname,attr,col,sort"";""trade,p,sym,1"")",1,1,write csv with bad column names +before,0,0,q,"`:tmp_dbw_badatt.csv 0: (""tabname,att,column,sort"";""trade,z,sym,1"")",1,1,write csv with invalid attribute value +before,0,0,q,"`:tmp_dbw_3col.csv 0: (""tabname,att,sort"";""trade,p,1"")",1,1,write csv missing a column +before,0,0,q,"`:tmp_dbw_empty.csv 0: enlist ""tabname,att,column,sort""",1,1,write header-only sort csv +before,0,0,q,"`:tmp_dbw_nodfl.csv 0: (""tabname,att,column,sort"";""trade,p,sym,1"")",1,1,write sort csv with no default row +before,0,0,q,"`:tmp_dbw_5col.csv 0: (""tabname,att,column,sort,extra"";""trade,p,sym,1,foo"")",1,1,write csv with an extra column +before,0,0,q,"`:tmp_dbw_reorder.csv 0: (""tabname,att,sort,column"";""trade,p,1,sym"";""trade,,0,time"";""default,p,1,sym"")",1,1,write valid csv with columns reordered +before,0,0,q,`:tmp_dbw_nohdr.csv 0: (),1,1,write a zero-line csv (no header) +before,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,init with no-op logger +before,0,0,q,cfg:([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,reusable valid config table (hand-built) +before,0,0,q,nodflcfg:([] tabname:enlist`trade; att:enlist`p; column:enlist`sym; sort:enlist 1b),1,1,reusable config with a trade row but no default + +comment,,,,,,,init - dependency injection validation (type errors / empty / null) +fail,0,0,q,dbwrite.init[(::)],1,1,init rejects :: as deps +fail,0,0,q,dbwrite.init[42],1,1,init rejects a non-dict deps value +fail,0,0,q,dbwrite.init[()!()],1,1,init rejects empty dict +fail,0,0,q,dbwrite.init[enlist[`other]!enlist mylog],1,1,init rejects dict missing log key +fail,0,0,q,dbwrite.init[enlist[`log]!enlist(::)],1,1,init rejects null log dep +fail,0,0,q,dbwrite.init[enlist[`log]!enlist 42],1,1,init rejects non-dict log value +fail,0,0,q,dbwrite.init[enlist[`log]!enlist(`info`warn!(mylog`info;mylog`warn))],1,1,init rejects log dict missing required key +true,0,0,q,"""di.dbwrite:""~11#@[{dbwrite.init[(::)]};(::);{x}]",1,1,error message is prefixed di.dbwrite +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,re-init with valid log dep + +comment,,,,,,,readcsv - stores a config from a csv into module state +run,0,0,q,dbwrite.readcsv `:tmp_dbw_valid.csv,1,1,readcsv stores config from a valid csv without error +true,0,0,q,3=count dbwrite.getconfig[],1,1,stored config has 3 rows +true,0,0,q,`tabname`att`column`sort~cols dbwrite.getconfig[],1,1,stored config has the correct column schema +true,0,0,q,(dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,stored config matches csv content +run,0,0,q,dbwrite.readcsv `:tmp_dbw_empty.csv,1,1,readcsv stores an empty config for a header-only csv +true,0,0,q,0=count dbwrite.getconfig[],1,1,empty csv produces empty stored config +run,0,0,q,dbwrite.readcsv `:tmp_dbw_reorder.csv,1,1,readcsv with reordered columns stores correct config +true,0,0,q,(`tabname`att`column`sort#dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,readcsv is column-order independent (0: names columns from the header) +true,0,0,q,(asc cols dbwrite.getconfig[])~`att`column`sort`tabname,1,1,readcsv reads all four columns regardless of input order +run,0,0,q,"dbwrite.readcsv ""tmp_dbw_valid.csv""",1,1,readcsv accepts a string file path +true,0,0,q,3=count dbwrite.getconfig[],1,1,string path readcsv stores the correct number of rows +true,0,0,q,(dbwrite.getconfig[])~([] tabname:`trade`trade`default; att:`p``p; column:`sym`time`sym; sort:101b),1,1,string path readcsv stores the correct config + +comment,,,,,,,readcsv - input and header validation failures +fail,0,0,q,dbwrite.readcsv `:nonexistent_file.csv,1,1,readcsv errors on a missing file +fail,0,0,q,dbwrite.readcsv 42,1,1,readcsv errors on a non-symbol non-string file argument +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_badcols.csv,1,1,readcsv rejects a csv with wrong column names +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_3col.csv,1,1,readcsv rejects a csv missing a required column +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_5col.csv,1,1,readcsv rejects a csv with an extra column +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_nohdr.csv,1,1,readcsv rejects a csv with no header row +fail,0,0,q,dbwrite.readcsv `:tmp_dbw_badatt.csv,1,1,readcsv rejects a csv with invalid attribute values + +comment,,,,,,,setconfig - stores a hand-built table into module state +run,0,0,q,dbwrite.setconfig[cfg],1,1,setconfig accepts a valid hand-built table +true,0,0,q,(dbwrite.getconfig[])~cfg,1,1,getconfig returns the table stored by setconfig +run,0,0,q,dbwrite.setconfig[0#cfg],1,1,setconfig accepts an empty (but correctly typed) table +true,0,0,q,0=count dbwrite.getconfig[],1,1,empty table stored correctly + +comment,,,,,,,setconfig - config validation (type errors / bad content) +fail,0,0,q,dbwrite.setconfig[42],1,1,setconfig rejects a non-table value +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; bad:enlist`p; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects unrecognised config columns +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`sym)],1,1,setconfig rejects a missing required config column +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`z; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects unknown attribute values +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`sym; sort:enlist 1)],1,1,setconfig rejects a non-boolean sort column +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`; att:enlist`p; column:enlist`sym; sort:enlist 1b)],1,1,setconfig rejects a null tabname in config +fail,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`p; column:enlist`; sort:enlist 1b)],1,1,setconfig rejects a null column in config + +comment,,,,,,,sort - happy path using stored config (set via setconfig or readcsv) +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to hand-built cfg +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort with a hand-built stored config and no partitions +run,0,0,q,dbwrite.readcsv `:tmp_dbw_valid.csv,1,1,set config via readcsv +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort with a csv-loaded stored config and no partitions + +comment,,,,,,,sort - tabname type validation +fail,0,0,q,dbwrite.sort[42;enlist`:/],1,1,sort errors on a non-symbol table name +true,0,0,q,"""di.dbwrite:""~11#@[{dbwrite.sort[42;enlist`:/]};(::);{x}]",1,1,non-symbol table name gives a di.dbwrite:-prefixed error + +comment,,,,,,,sort - edge cases (empty config / null att / empty dirs / atom dir) +run,0,0,q,dbwrite.setconfig[0#cfg],1,1,set empty config +true,0,0,q,()~dbwrite.sort[`trade;enlist`:/],1,1,empty config yields no work and returns () +run,0,0,q,dbwrite.setconfig[([] tabname:enlist`trade; att:enlist`; column:enlist`sym; sort:enlist 0b)],1,1,set config with a null (no) attribute row +run,0,0,q,dbwrite.sort[`trade;enlist`:/],1,1,sort accepts a null (no) attribute row +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;()],1,1,sort handles an empty partition list without error +run,0,0,q,dbwrite.sort[`trade;`:/],1,1,sort accepts a single atom hsym dir not just a list + +comment,,,,,,,sort - params resolution (table-specific / default / none / default fallback) +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;enlist`:/],1,1,sort uses the table-specific params row +run,0,0,q,dbwrite.sort[`other;enlist`:/],1,1,sort falls back to the default params row +run,0,0,q,dbwrite.setconfig[nodflcfg],1,1,set config to nodflcfg (no default row) +true,0,0,q,()~dbwrite.sort[`other;enlist`:/],1,1,sort returns () when no params found and no default row +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset module state (clears sortconfig to (::)) +run,0,0,q,dbwrite.sort[`anytable;()],1,1,sort with unset sortconfig falls back to the built-in default config + +comment,,,,,,,sort - on-disk end-to-end (explicit config sorts by sym and applies p#) +run,0,0,q,`:dbw_sort_tp/.d set `sym`price,1,1,write column order file +run,0,0,q,`:dbw_sort_tp/sym set `IBM`AAPL`MSFT,1,1,write unsorted sym column +run,0,0,q,`:dbw_sort_tp/price set 200 100 300f,1,1,write price column +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg before sort +run,0,0,q,dbwrite.sort[`trade;`:dbw_sort_tp/],1,1,sort the trade partition by its config +true,0,0,q,`AAPL`IBM`MSFT~exec sym from get `:dbw_sort_tp/,1,1,sym sorted ascending on disk +true,0,0,q,`p=attr get `:dbw_sort_tp/sym,1,1,p attribute applied to sym on disk + +comment,,,,,,,sort - on-disk end-to-end (default config sorts by time) +run,0,0,q,`:dbw_def_tp/.d set `time`sym,1,1,write column order file +run,0,0,q,`:dbw_def_tp/time set 2024.01.01D09:00 2024.01.01D08:00,1,1,write unsorted time column +run,0,0,q,`:dbw_def_tp/sym set `IBM`AAPL,1,1,write sym column +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset sortconfig to (::) for default-fallback test +run,0,0,q,dbwrite.sort[`anytable;`:dbw_def_tp/],1,1,sort using the built-in default config +true,0,0,q,(asc exec time from get `:dbw_def_tp/)~exec time from get `:dbw_def_tp/,1,1,time sorted ascending by the default config + +comment,,,,,,,sort - multiple partition dirs in one call and a non-fatal partition failure +run,0,0,q,`:dbw_md1/.d set `sym`px,1,1,write md1 column order +run,0,0,q,`:dbw_md1/sym set `c`a`b,1,1,write md1 unsorted sym +run,0,0,q,`:dbw_md1/px set 1 2 3f,1,1,write md1 price +run,0,0,q,`:dbw_md2/.d set `sym`px,1,1,write md2 column order +run,0,0,q,`:dbw_md2/sym set `b`c`a,1,1,write md2 unsorted sym +run,0,0,q,`:dbw_md2/px set 1 2 3f,1,1,write md2 price +run,0,0,q,dbwrite.setconfig[cfg],1,1,set config to cfg +run,0,0,q,dbwrite.sort[`trade;(`:dbw_md1/;`:dbw_md2/)],1,1,sort two partition dirs in one call +true,0,0,q,`a`b`c~exec sym from get `:dbw_md1/,1,1,first partition sorted +true,0,0,q,`a`b`c~exec sym from get `:dbw_md2/,1,1,second partition sorted +run,0,0,q,`:dbw_good/.d set `sym`px,1,1,write good-partition column order +run,0,0,q,`:dbw_good/sym set `c`a`b,1,1,write good-partition unsorted sym +run,0,0,q,`:dbw_good/px set 1 2 3f,1,1,write good-partition price +run,0,0,q,dbwrite.sort[`trade;(`:dbw_good/;`:/)],1,1,sort a good dir alongside a bad one +true,0,0,q,`a`b`c~exec sym from get `:dbw_good/,1,1,good partition still sorted despite the bad one failing + +comment,,,,,,,applyattr - applies an attribute on disk and is a no-op for non-attributes +run,0,0,q,`:dbw_attr_tp/.d set `sym`price,1,1,write column order file +run,0,0,q,`:dbw_attr_tp/sym set `IBM`AAPL`MSFT,1,1,write sym column +run,0,0,q,`:dbw_attr_tp/price set 200 100 300f,1,1,write price column +run,0,0,q,dbwrite.applyattr[`:dbw_attr_tp/;`sym;`p],1,1,apply p attr to sym +true,0,0,q,`p=attr get `:dbw_attr_tp/sym,1,1,p attribute applied to sym on disk +run,0,0,q,dbwrite.applyattr[`:dbw_attr_tp/;`price;`z],1,1,applyattr is a silent no-op for an invalid attribute +true,0,0,q,`=attr get `:dbw_attr_tp/price,1,1,invalid attribute leaves the column unattributed + +comment,,,,,,,savedown - write an in-memory table to an hdb partition then sort it +run,0,0,q,"sdtbl:([]time:2024.01.01D09:00 2024.01.01D08:00;sym:`IBM`AAPL;price:100 200f)",1,1,unsorted test table +run,0,0,q,dbwrite.init[enlist[`log]!enlist mylog],1,1,reset sortconfig to (::) for default-config savedown +run,0,0,q,dbwrite.savedown[`:dbw_hdb;2024.01.01;`trade;sdtbl],1,1,savedown with the default config (sort by time) +run,0,0,q,"sdpath:` sv (.Q.par[`:dbw_hdb;2024.01.01;`trade];`)",1,1,resolve the partition path +true,0,0,q,2=count get sdpath,1,1,two rows written to the partition +true,0,0,q,(asc exec time from get sdpath)~exec time from get sdpath,1,1,partition sorted by time after savedown +true,0,0,q,0type deps; + '"di.eodtime: deps must be a dict with `log key"]; + if[not `log in key deps; + '"di.eodtime: log dependency is required; pass at minimum `info!{[c;m]} keyed on `log"]; + if[99h<>type deps`log; + '"di.eodtime: log value must be a dict; pass at minimum `info!{[c;m]}"]; + if[not `info in key deps`log; + '"di.eodtime: log dict must have at minimum an `info key; got: ",(", " sv string key deps`log)]; + .z.m.loginfo:(deps`log)`info; + .z.m.rolltimezone:$[`rolltimezone in key deps;deps`rolltimezone;`$"GMT"]; + .z.m.datatimezone:$[`datatimezone in key deps;deps`datatimezone;`$"GMT"]; + .z.m.rolltimeoffset:$[`rolltimeoffset in key deps;deps`rolltimeoffset;0D00:00:00.000]; + .z.m.dailyadj:getdailyadjustment[]; + .z.m.d:getday[.z.p]; + .z.m.nextroll:getroll[.z.p]; + .z.m.loginfo[`eodtime;"initialised with rolltimezone=",string[.z.m.rolltimezone]," datatimezone=",string[.z.m.datatimezone]," rolltimeoffset=",string .z.m.rolltimeoffset]; + }; \ No newline at end of file diff --git a/di/eodtime/init.q b/di/eodtime/init.q new file mode 100644 index 00000000..027c0e12 --- /dev/null +++ b/di/eodtime/init.q @@ -0,0 +1,7 @@ +/ end-of-day time management - date resolution, roll scheduling and data timestamp adjustment + +tz:use`di.tz + +\l ::eodtime.q + +export:([init;getd;getnextroll;getdailyadj;getroll;getdailyadjustment;setnextroll;setdailyadj;setd]) diff --git a/di/eodtime/test.csv b/di/eodtime/test.csv new file mode 100644 index 00000000..1fdd0e36 --- /dev/null +++ b/di/eodtime/test.csv @@ -0,0 +1,92 @@ +action,ms,bytes,lang,code,repeat,minver,comment +comment,,,,,,,setup - load module and initialise with gmt defaults and log dep +before,0,0,q,eodtime:use`di.eodtime,1,1,load di.eodtime module +before,0,0,q,logdep:`info`warn`error!({[c;m]};{[c;m]};{[c;m]}),1,1,silent no-op log mock - binary {[c;m]} matches the log contract +before,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"GMT";0D00:00:00.000;logdep)],1,1,init with gmt defaults and log dep + +comment,,,,,,,getd +true,0,0,q,-14h=type eodtime.getd[],1,1,getd returns date type + +comment,,,,,,,getnextroll +true,0,0,q,-12h=type eodtime.getnextroll[],1,1,getnextroll returns timestamp type +true,0,0,q,eodtime.getnextroll[]>.z.p,1,1,getnextroll returns future timestamp + +comment,,,,,,,getdailyadj +true,0,0,q,-16h=type eodtime.getdailyadj[],1,1,getdailyadj returns timespan type +true,0,0,q,0D=eodtime.getdailyadj[],1,1,getdailyadj returns 0D for gmt datatimezone + +comment,,,,,,,getdailyadjustment +true,0,0,q,-16h=type eodtime.getdailyadjustment[],1,1,getdailyadjustment returns timespan type +true,0,0,q,0D=eodtime.getdailyadjustment[],1,1,getdailyadjustment returns 0D for gmt datatimezone + +comment,,,,,,,getroll - gmt zero offset +true,0,0,q,-12h=type eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns timestamp type +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns next midnight for gmt zero offset + +comment,,,,,,,getroll - gmt with rolltimeoffset +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"GMT";0D17:00:00.000;logdep)],1,1,reinit with 5pm gmt roll time +true,0,0,q,2025.01.01D17:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns todays 5pm when before roll time +true,0,0,q,2025.01.02D17:00:00.000000000=eodtime.getroll[2025.01.01D18:00:00.000000000],1,1,getroll returns next day 5pm when past roll time + +comment,,,,,,,setnextroll +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"GMT";0D00:00:00.000;logdep)],1,1,reinit with gmt defaults +run,0,0,q,eodtime.setnextroll[2025.06.01D00:00:00.000000000],1,1,setnextroll does not error +true,0,0,q,2025.06.01D00:00:00.000000000=eodtime.getnextroll[],1,1,getnextroll reflects setnextroll + +comment,,,,,,,setdailyadj +run,0,0,q,eodtime.setdailyadj[0D01:00:00.000000000],1,1,setdailyadj does not error +true,0,0,q,0D01:00:00.000000000=eodtime.getdailyadj[],1,1,getdailyadj reflects setdailyadj + +comment,,,,,,,setd +run,0,0,q,eodtime.setd[2025.06.01],1,1,setd does not error +true,0,0,q,2025.06.01=eodtime.getd[],1,1,getd reflects setd + +comment,,,,,,,utc-equivalent timezone shortcuts (GMT/UTC/Etc/GMT) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"UTC";`$"Etc/GMT";0D00:00:00.000;logdep)],1,1,reinit with UTC/Etc/GMT shortcuts +true,0,0,q,0D=eodtime.getdailyadjustment[],1,1,Etc/GMT datatimezone returns 0D without lookup +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,UTC rolltimezone returns correct roll + +comment,,,,,,,non-gmt rolltimezone (Europe/London) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"Europe/London";`$"GMT";0D00:00:00.000;logdep)],1,1,reinit with London rolltimezone +true,0,0,q,-12h=type eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll returns timestamp type for non-gmt rolltimezone +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,getroll correct for london winter (utc+0) +true,0,0,q,2025.07.01D23:00:00.000000000=eodtime.getroll[2025.07.01D12:00:00.000000000],1,1,getroll correct for london summer (utc+1 - midnight local is 23:00 utc) + +comment,,,,,,,getroll - dst transition overnight (europe/london clocks go back 2025.10.26 01:00 utc) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"Europe/London";`$"GMT";0D05:00:00.000;logdep)],1,1,reinit with 5am london roll time +true,0,0,q,2025.10.26D05:00:00.000000000=eodtime.getroll[2025.10.25D12:00:00.000000000],1,1,rollover uses tomorrows post-transition offset (gmt) not todays bst offset + +comment,,,,,,,non-gmt datatimezone (Europe/London - DST aware) +run,0,0,q,eodtime.init[`rolltimezone`datatimezone`rolltimeoffset`log!(`$"GMT";`$"Europe/London";0D00:00:00.000;logdep)],1,1,reinit with London datatimezone +true,0,0,q,-16h=type eodtime.getdailyadjustment[],1,1,Europe/London datatimezone returns timespan type +true,0,0,q,0D<=eodtime.getdailyadjustment[],1,1,Europe/London datatimezone returns non-negative offset + +comment,,,,,,,init - empty config (just log dep) applies all defaults +run,0,0,q,eodtime.init[enlist[`log]!enlist logdep],1,1,init with just log dep applies all defaults +true,0,0,q,0D=eodtime.getdailyadj[],1,1,default datatimezone (GMT) produces zero daily adjustment +true,0,0,q,0D=eodtime.getdailyadjustment[],1,1,default datatimezone (GMT) produces zero computed adjustment +true,0,0,q,2025.01.02D00:00:00.000000000=eodtime.getroll[2025.01.01D12:00:00.000000000],1,1,default rolltimeoffset (0D) gives midnight roll + +comment,,,,,,,init - non-dict configs throws +fail,0,0,q,eodtime.init[(::)],1,1,errors when configs is not a dictionary + +comment,,,,,,,error cases +fail,0,0,q,eodtime.getroll[`notvalid],1,1,getroll throws on non-timestamp input + +comment,,,,,,,init - dep validation +fail,0,0,q,eodtime.init[()!()],1,1,errors when log dependency is missing +fail,0,0,q,eodtime.init[enlist[`log]!enlist 42],1,1,errors when log value is not a dictionary +run,0,0,q,eodtime.init[enlist[`log]!enlist `info`warn!(logdep`info;logdep`warn)],1,1,init succeeds with info and warn but no error key - only info is required +run,0,0,q,.test.err:@[{eodtime.init[()!()]};(::);{x}],1,1,capture error string from init with missing log dep +true,0,0,q,.test.err like "di.eodtime:*",1,1,error is prefixed di.eodtime: + +comment,,,,,,,init - accepts any dict already shaped to the binary {[c;m]} contract (e.g. di.log's logdict) +run,0,0,q,sixlvllog:`trace`debug`info`warn`error`fatal!({[c;m]};{[c;m]};{[c;m]};{[c;m]};{[c;m]};{[c;m]}),1,1,hand-built six-level dict matching di.log's logdict shape +run,0,0,q,eodtime.init[`log`rolltimezone!(sixlvllog;`$"GMT")],1,1,superset log dict accepted as-is - only info/warn/error are used + +comment,,,,,,,log call verification - capturing logger sees init message with binary {[c;m]} args +run,0,0,q,caplog:`info`warn`error!({[c;m] `.test.cap upsert(`info;m)};{[c;m] `.test.cap upsert(`warn;m)};{[c;m] `.test.cap upsert(`error;m)}),1,1,capturing log mock - binary {[c;m]} +run,0,0,q,.test.cap:([] fn:`symbol$();msg:()),1,1,initialise log capture table +run,0,0,q,eodtime.init[enlist[`log]!enlist caplog],1,1,init with capturing logger +true,0,0,q,1=count select from .test.cap where fn=`info,1,1,one info entry captured on init +true,0,0,q,any (.test.cap`msg) like "initialised*",1,1,log message confirms initialisation diff --git a/di/log/init.q b/di/log/init.q new file mode 100644 index 00000000..cd68a4e4 --- /dev/null +++ b/di/log/init.q @@ -0,0 +1,4 @@ +// structured logger - default log dependency for di.* modules +\l ::log.q + +export:([createlog;logdict;trace;debug;info;warn;error;fatal]) diff --git a/di/log/log.md b/di/log/log.md new file mode 100644 index 00000000..ae955c88 --- /dev/null +++ b/di/log/log.md @@ -0,0 +1,193 @@ +# Log + +`log.q` is the default logging implementation for `di.*` modules. It writes formatted lines to +stdout and satisfies the log dependency contract expected by modules such as `di.email`. It also +provides `createlog`, a factory for rich structured logger instances with level filtering, +multiple output sinks, and configurable format templates. + +## Usage + +```q +logger:use`di.log +logger.trace[`mymodule;"entering function"] +logger.debug[`mymodule;"value is 42"] +logger.info[`mymodule;"starting up"] +logger.warn[`mymodule;"disk usage above 80%"] +logger.error[`mymodule;"connection failed"] +logger.fatal[`mymodule;"unrecoverable error, shutting down"] +``` + +Output format: + +``` +2026-04-09T12:00:00.000000000 [TRACE] [mymodule] entering function +2026-04-09T12:00:00.001000000 [DEBUG] [mymodule] value is 42 +2026-04-09T12:00:00.002000000 [INFO] [mymodule] starting up +2026-04-09T12:00:00.003000000 [WARN] [mymodule] disk usage above 80% +2026-04-09T12:00:00.004000000 [ERROR] [mymodule] connection failed +2026-04-09T12:00:00.005000000 [FATAL] [mymodule] unrecoverable error, shutting down +``` + +## Injecting into other modules + +All `di.*` modules that accept a log dependency expect a dictionary with keys `` `info`warn`error ``, each a function with signature `{[ctx;msg]}`. + +```q +logger:use`di.log +logdep:`info`warn`error!(logger.info;logger.warn;logger.error) + +email:use`di.email +email.init[enlist[`log]!enlist logdep] +``` + +You can extend the injected dictionary with `trace`, `debug`, and `fatal` for modules that support them: + +```q +logdep:`trace`debug`info`warn`error`fatal!(logger.trace;logger.debug;logger.info;logger.warn;logger.error;logger.fatal) +``` + +`di.log` has no dependencies of its own, so there's no `init` to call on it first — use +`logdict` to skip building the dict by hand. It's the dependency already wrapped as +`` `log!enlist logdict ``, ready to pass straight into any `di.*` module's `init`: + +```q +mylog:use`di.log +email:use`di.email +email.init[mylog.logdict] +``` + +## createlog + +`createlog` is a factory that returns an independent logger instance with level filtering, multiple output sinks, and configurable format templates. Each call to `createlog` produces a separate instance with its own state. + +Instance-level functions (`trace`..`fatal`) take the same `{[ctx;msg]}` signature as the plain top-level functions, so an instance satisfies the log dependency contract directly. None of the built-in format templates have a slot for `ctx`, so it's folded into the message text as `[ctx] msg`. + +```q +logger:use`di.log +mylog:logger.createlog[] + +mylog.setlvl `warn / suppress trace, debug, info +mylog.info[`mymodule;"this is suppressed"] / returns () silently +mylog.warn[`mymodule;"this appears"] + +mylog.setfmt `syslog / switch to syslog format +mylog.addfmt[`compact;"$l $m"] / add a custom format +mylog.setfmt `compact + +mylog.add[2i;`error`fatal] / add stderr sink for error and fatal +mylog.remove[1i;`trace] / remove stdout from trace level +``` + +### Sinks + +A sink is a handle (integer file descriptor or function) passed to `add`. If a function is provided it is called with the formatted line string. Built-in handles follow standard q conventions: `1i` is stdout, `2i` is stderr. Since a sink can be any open handle, `hopen` a real file to log to disk: + +```q +fh:hopen`:/path/to/app.log +mylog.add[fh;lvls] / route every level to the file +mylog.remove[1i;lvls] / optionally drop the default stdout sink +mylog.info[`mymodule;"this now goes to app.log"] +``` + +Or use a plain function sink to capture output in memory: + +```q +buf:(); +capture:{[msg] buf,:enlist msg}; / function sink - captures output +mylog.add[capture;`info`warn`error] +mylog.info[`mymodule;"captured"] +buf / ("2026-...captured\n") +mylog.remove[capture;`info] +``` + +### Format templates + +Three built-in formats are available: + +| Name | Template | Example output | +|---|---|---| +| `basic` (default) | `$p $l PID[$i] HOST[$h] $m` | `2026-04-09T12:00:00.000000000 INFO PID[1234] HOST[myhost] message` | +| `syslog` | `<$s> $m` | `<6> message` | +| `raw` | `$m` | `message` | + +Template variables: `$p` timestamp, `$l` level, `$i` PID, `$h` hostname, `$m` message, `$s` syslog severity number. + +## API + +### `trace` +Parameters: `[ctx; msg]` + +Write a trace-level message to stdout. + +- `ctx` — symbol context tag (e.g. `` `mymodule ``) +- `msg` — string message + +### `debug` +Parameters: `[ctx; msg]` + +Write a debug-level message to stdout. + +### `info` +Parameters: `[ctx; msg]` + +Write an info-level message to stdout. + +### `warn` +Parameters: `[ctx; msg]` + +Write a warning-level message to stdout. + +### `error` +Parameters: `[ctx; msg]` + +Write an error-level message to stdout. + +### `fatal` +Parameters: `[ctx; msg]` + +Write a fatal-level message to stdout. + +### `logdict` +A dictionary, not a function. + +`` `log!enlist(`trace`debug`info`warn`error`fatal!(...))`` — the log dependency dict pre-wrapped +exactly as any `di.*` module's `init` expects `deps`, so it can be passed straight through +without building it by hand. It includes all six levels, a superset of the `info`/`warn`/`error` +contract minimum, since the underlying `createlog[]` instance computes them anyway — a module +that supports the optional extended levels gets them for free. Backed by its own independent +`createlog[]` instance, separate from the plain `trace`..`fatal` functions above. + +### `createlog` +Parameters: none + +Returns an independent logger instance as a dictionary of functions. Each call returns a new instance with isolated state. + +Returned keys: `` `trace`debug`info`warn`error`fatal`add`remove`setfmt`getfmt`addfmt`setlvl`getlvl `` + +| Function | Parameters | Description | +|---|---|---| +| `trace`..`fatal` | `[ctx;msg]` | Write a message at the given level (filtered by active level) | +| `setlvl` | `[lvl]` | Set minimum level; one of `` `trace`debug`info`warn`error`fatal `` | +| `getlvl` | `[_]` | Return current minimum level | +| `setfmt` | `[name]` | Switch to a named format template | +| `getfmt` | `[_]` | Return current format name | +| `addfmt` | `[name;template]` | Register a new named format template | +| `add` | `[handle;lvls]` | Add a sink for one or more levels; returns the handle | +| `remove` | `[handle;lvl]` | Remove a sink from a level. `handle` may be a bare handle (removes every sink registered with it) or a `(handle;fn)` pair matching what was passed to `add` (removes only that exact sink) | + +## Log dependency contract + +The log dependency contract used across `di.*` modules requires a dictionary: + +```q +`info`warn`error!({[ctx;msg] ...};{[ctx;msg] ...};{[ctx;msg] ...}) +``` + +`di.log` satisfies this contract. You can also supply any custom implementation with the same signatures. + +## Testing + +```q +q)k4unit:use`di.k4unit +q)k4unit.moduletest`di.log +``` diff --git a/di/log/log.q b/di/log/log.q new file mode 100644 index 00000000..c923f7c3 --- /dev/null +++ b/di/log/log.q @@ -0,0 +1,193 @@ +/ structured logger for di.* modules +/ provides info, warn, error, trace, debug, fatal with signature {[ctx;msg]} +/ ctx is a symbol context tag, msg is a string +/ also provides createlog, a factory for rich structured logger instances + +/ os-aware newline +nl:$[.z.o in `w32`w64;"\r\n";"\n"]; + +/ log levels in priority order +lvls:`trace`debug`info`warn`error`fatal; + +/ syslog severity per level (rfc5424) +sysloglvl:lvls!7 7 6 4 3 2i; + +/ built-in format templates for createlog instances +fmts:`basic`syslog`raw!("$p $l PID[$i] HOST[$h] $m";"<$s> $m";"$m"); + +/ format pattern handlers for createlog instances; each takes {[level;msg]} +/ level is an uppercase string (e.g. "INFO"), msg is the formatted message string +pattern:"plihms~"!( + {[x;y] string .z.p}; + {[x;y] x}; + {[x;y] string .z.i}; + {[x;y] string .z.h}; + {[x;y] y}; + {[x;y] string sysloglvl`$lower x}; + {[x;y] "$"}); + +/ split a format template on delimiter, returning (textparts; substitutionfunctions) +/ escaped delimiter (e.g. $$) is replaced by $~ which resolves to literal $ +fmtprep:{[del;rep;fmt] + fmt:ssr[fmt;del,del;del,"~"]; + parts:del vs fmt; + fns:rep@first each 1_parts; + (enlist[first parts],1_/:1_parts;fns) + }; + +/ apply a prepared format returning the assembled line string +/ level is an uppercase string, msg is the formatted message string +fmtapply:{[prep;level;msg] + textparts:prep 0; + fns:prep 1; + vals:fns .\:(level;msg); + raze first[textparts],vals,'1_textparts + }; + +/ apply printf-style variable substitution to a message +/ msg is a plain string or (fmtstring;arg1;arg2;...) +/ %s converts to string, %r uses .Q.s1, %% becomes a literal percent +fmtmsg:{[msg] + $[10h=abs type msg; + msg; + [fmt:first msg; + args:1_msg; + if[not 10h=abs type fmt;'"format string must be a string"]; + fmt:ssr[fmt;"%%";"\000"]; + parts:"%" vs fmt; + nspecs:count[parts]-1; + if[nspecs<>count args;'`$"expected ",string[nspecs]," argument(s) for format string, got ",string count args]; + subs:"sr"!({$[10h=abs type x;x;-11h=type x;string x;'`type]};.Q.s1); + result:first[parts],raze {[subs;args;parts;i] + part:parts[1+i]; + code:first part; + rest:1_part; + if[not code in key subs;'`$"unsupported format char: ",enlist code]; + (subs[code] args[i]),rest + }[subs;args;parts;] each til count[parts]-1; + ssr[result;"\000";"%"] + ] + ]}; + +/ format and write a log line to stdout using the dependency-contract format +logline:{[level;ctx;msg] + -1 (string .z.p)," [",level,"] [",string[ctx],"] ",msg; + }; + +/ instance counter and per-instance state; always accessed via .z.m so nested closures (e.g. +/ createlog's returned methods) reliably resolve to this module's namespace. .z.M does not +/ resolve correctly inside nested closures here and must not be used for this state. +i:0; +inst:()!(); + +/ factory helper: returns a {[ctx;msg]} log function for the given level and instance +/ each entry in sink is a (handle;sender) pair; sender is called with the formatted text +makelevel:{[id;gv;lvl] + {[id;gv;lvl;ctx;msg] + if[(lvls?lvl) version strings +true,0,0,q,expdeps~(tml.parsetoml fdeps)`dependencies,1,1,a deps.toml [dependencies] table parses to the symbol!version-string dict di.depcheck readdeps expects +true,0,0,q,11h=type key (tml.parsetoml fdeps)`dependencies,1,1,dependency names are symbols (quoted dotted keys unquoted) +true,0,0,q,all 10h=type each value (tml.parsetoml fdeps)`dependencies,1,1,version values stay strings (policy-free) for meetsmin comparison +comment,,,,,,,getapimeta +true,0,0,q,"(asc (key tml) except `getapimeta)~asc exec name from tml.getapimeta[]",1,1,getapimeta documents exactly the callable exports (plumbing omitted) +true,0,0,q,not `getapimeta in exec name from tml.getapimeta[],1,1,getapimeta does not register itself (plumbing not in the api) +true,0,0,q,`name`public`descrip`params`return~cols tml.getapimeta[],1,1,getapimeta rows carry the registry columns +after,0,0,q,teardowntoml[],1,1,remove the temp fixture dir diff --git a/di/toml/test.q b/di/toml/test.q new file mode 100644 index 00000000..2fcf18b3 --- /dev/null +++ b/di/toml/test.q @@ -0,0 +1,30 @@ +/ di.toml test fixtures (loaded by test.csv). di.toml is pure - no mocks needed; the TOML text is +/ defined here as q string literals (far cleaner than CSV-escaping all the quotes inline) plus a +/ small temp .toml file fixture for parsefile. + +fflat:"a = 1\nb = 2.5\nc = true\nd = false\ns = \"hello\""; +fqkey:"\"my key\" = 42"; +fcomments:"# whole line\na = 1 # trailing\ns = \"a # b\" # real comment"; +fsection:"top = 1\n[sec]\nk = 10\nname = \"rdb\""; +farrays:"nums = [1, 2, 3]\nstrs = [\"xx\", \"yy\"]\nempty = []"; +fescapes:"s = \"a\\nb\\tc\\\"d\""; +fblank:"\n \n# only a comment\n"; +fqdotkey:"\"a.b\" = 1"; / a QUOTED key with a dot - a literal key, allowed +expesc:"a\nb\tc\"d"; / expected unescaped value of fescapes' s +fbs:"p = \"a\\\\nb\""; / TOML p = "a\\nb" (escaped backslash then n) +expbs:"a\\nb"; / expected: a, backslash, n, b (4 chars) - NOT a newline +femptystr:"k = \"\""; / TOML k = "" (a legitimate empty string) +finvesc:"s = \"a\\xb\""; / TOML s = "a\xb" (\x is not a valid escape) +fescq:"d = \"a\\\"b\" # c"; / TOML d = "a\"b" # c (escaped quote in value + comment) +expescq:"a\"b"; / expected value of d: a, ", b (comment stripped) +fescarr:"a = [\"x\\\"y\", \"zz\"]"; / TOML a = ["x\"y", "zz"] (escaped quote in an array elem) +expescarr:("x\"y";"zz"); / expected: two elements x"y and zz (comma not mis-split) +fqsec:"[\"a.b\"]\nx = 1"; / TOML ["a.b"] - a QUOTED section name (dot is literal) +ftab:"s = \"x\ty\""; / TOML s = "xy" (a literal tab inside the string) +exptab:"x\ty"; / expected: x, tab, y - the tab preserved, not spaced +fdeps:"# peer deps\n[dependencies]\n\"di.servers\" = \"0.3.0\" # injected\n\"di.dbwrite\" = \"0.1.0\""; / a di.depcheck deps.toml +expdeps:`di.servers`di.dbwrite!("0.3.0";"0.1.0"); / di.depcheck's readdeps expects [dependencies] -> symbol!version-string + +TDIR:"/tmp/ditomltest"; +setuptoml:{[] system "mkdir -p ",TDIR; (`$":",TDIR,"/x.toml") 0: ("dir = \":appdb\"";"rows = 100");}; +teardowntoml:{[] system "rm -rf ",TDIR;}; diff --git a/di/toml/toml.md b/di/toml/toml.md new file mode 100644 index 00000000..176e0e51 --- /dev/null +++ b/di/toml/toml.md @@ -0,0 +1,104 @@ +# di.toml + +A small, scoped-down TOML parser for the modular TorQ world. It reads settings `.toml` files (and +in-memory TOML text) into q dicts. Its reason for existing is to back di.config's `.toml` +resolution tier — di.config lazily loads di.toml only when a `.toml` file is actually parsed. + +**Pure module** — no `init`, no logger, no injected dependencies. di.config invokes it *during +config resolution*, before any logger exists, so parse errors are **signalled** with a clear +`'di.toml: …'` message rather than logged. + +**Fail-loud** — because this is a *shared* utility (any module that parses a file may use it), +di.toml never silently returns nulls or garbage for input it can't correctly handle: a missing +file, an unparseable value (bare word / datetime), a dotted key, or a malformed/array-of-tables +section header all **signal**. A caller that treats a *missing file* as acceptable (a config +cascade probing optional tiers) must **guard existence itself** before calling — the way di.config +does (`if[0=count key hsym`$path; :()!()]`) — di.toml does not paper over it. + +## Scope (v1) + +Supported — what real settings files need: + +- `key = value` pairs; keys bare or double-quoted (`"my key" = 1`) +- `#` comments, whole-line or trailing — **quote-aware**, so a `#` inside a `"…"` string is not a + comment +- one level of `[section]` nesting — a section becomes a **sub-dict** keyed by the section name +- scalar values: double-quoted strings (`\"` `\\` `\n` `\t` escapes), integers, floats, + `true`/`false`, and flat arrays of any of those + +Deliberately **not** supported (out of scope for settings): nested inline tables (`{a=1,b=2}`), +dotted keys (`a.b.c`), single-quoted literal strings, multi-line strings, datetimes, and +array-of-tables (`[[x]]`). + +## Type policy + +- **Strings are policy-free.** TOML has no symbol type, so every TOML string parses to a q **char + string** (`10h`), never a symbol. Callers that want a symbol coerce at the point of use (`` `$ `` + is a no-op on an already-a-symbol value, so the same consumer code works for `.q` settings too). + This is the contract di.config and its consumers rely on. +- **Integers parse to `long`** (matching what `value` gives a `.q` settings line like `rows:100`), + floats to `float`, `true`/`false` to `boolean`. + +## Exported functions + +| Function | Signature | Description | +|---|---|---| +| `parsetoml` | `parsetoml[text]` | Parse a TOML string into a dict (one level of `[section]` nesting). Blank and comment-only lines contribute nothing. | +| `parsefile` | `parsefile[path]` | Read and parse a `.toml` file into a dict. A **missing file signals** — a caller that treats missing as acceptable (a config cascade) guards existence itself first (as di.config does). | +| `getapimeta` | `getapimeta[]` | This module's api metadata — one row per **callable** function (`parsetoml`, `parsefile`); `getapimeta` itself is plumbing and is deliberately not listed. For `di.torq` to register with `di.api`. | + +Export is conservative — the internal helpers (`trimstr`, `isquoted`, `instrmask`, `firstunquoted`, +`stripcomment`, `splitassign`, `unquotekey`, `parsekey`, `unescape`, `parsescalar`, `splitcommas`, +`parsevalue`, `sectname`, `addkv`, `addline`) are not exported. No `version` export / `VERSION` file yet — deferred +to the di.depcheck rollout, as in di.config/di.servers. + +## di.config integration contract + +di.config's `parsefile` delegates a `.toml` path via `` (use`di.toml)[`parsefile] path ``, expecting +a **flat dict** of `setting → value` back (section values are sub-dicts). This module satisfies that +directly. Once di.toml is on `QPATH` alongside di.config, the `.toml` half of di.config's cascade +(and `.toml > .q` within a tier) works, and di.config's guard-rail error (`the di.toml module was +not found on QPATH`) no longer fires. + +## Out-of-scope constructs are rejected, not mis-parsed + +Anything the scoped grammar can't correctly represent **signals** rather than silently producing +wrong data: + +- a bare/unquoted non-numeric value (a word, a datetime) +- an **invalid bare key or section name** — one with a space, `:`, or any char outside `A-Za-z0-9_-` + (this is what makes most *non-TOML* lines that happen to contain `=` — a shell `export FOO=5`, a q + `x:a=5` — fail loud instead of parsing to a bogus symbol key). Keys and section names are held to + the *same* rule (a shared `parsekey`) +- an **empty key or section name** (`= 5`, `[]`) +- a **dotted key or section** (`a.b`, `[a.b]` — nesting) — but a *quoted* `"a.b"` (key or section) is + a legitimate literal name and is kept, unquoted +- a **duplicate key / section** (or a key/section-name clash) — a TOML error, not silent last-wins +- an **empty (missing) value** `k =`, and an **unterminated / malformed array** (`[` without a `]`) +- an **unknown or dangling string escape** (`\x`, a trailing `\`) +- an **array-of-tables** `[[x]]`, or a malformed section header (missing `]`) +- a non-char input, or a 1-char input (a char atom) — handled/validated at the `parsetoml` entry so + it never surfaces as a cryptic `'type` + +This is what makes di.toml safe as a shared parser — a consumer feeding a richer or malformed file +gets a clear `'di.toml: …'` error, not quiet garbage. + +## Known limitations + +None that mis-parse. Everything outside the supported grammar — including TOML number forms not +covered (underscored `1_000`, hex/octal/binary `0x`/`0o`/`0b`, `inf`/`nan`) — is **rejected with a +clear signal**, never silently mis-parsed. The scope is deliberately the settings-file subset; widen +it (and add tests) if a real settings file needs more. + +## Tests + +```q +k4unit:use`di.k4unit +k4unit.moduletest`di.toml +``` + +`test.csv` (+ `test.q` fixtures) covers scalars and their types, policy-free strings, quoted keys, +quote-aware comments, one-level sections, arrays (int/string/empty), the `\n`/`\t`/`\"` escapes, +blank/comment-only input, a missing `=` signalling, `parsefile` (present + missing file), and +`getapimeta`. TOML text lives in `test.q` as q string literals (cleaner than CSV-escaping the quotes +inline). Runs from the repo root (`test.q` is loaded relative). diff --git a/di/toml/toml.q b/di/toml/toml.q new file mode 100644 index 00000000..e8a096b7 --- /dev/null +++ b/di/toml/toml.q @@ -0,0 +1,128 @@ +/ scoped-down TOML parser for the modular torq world - reads settings .toml files (and text) into a +/ q dict. supports: key=value (bare or "quoted" keys), quote-aware # comments, one level of [section] +/ nesting, and scalars (quoted strings, integer->long, float, true/false, flat arrays). not supported: +/ inline tables, dotted keys, single-quoted/multi-line strings, datetimes, array-of-tables. +/ fail-loud: anything it cannot correctly parse signals 'di.toml: ...', never silent garbage. +/ strings parse to q char strings, never symbols (TOML has none) - callers coerce with `$ at use. +/ pure: no init/logger/deps - di.config loads it during config resolution, before a logger exists. +/ parsefile signals on a MISSING file; a caller for whom that is acceptable (a cascade probing +/ optional tiers) guards existence itself first, as di.config does: if[0=count key hsym`$path;:()!()]. +/ reserved-word trap: cut/trim/parse/ss/sv/vs/ssr are builtins - hence trimstr/parsetoml/etc. + +keychars:.Q.a,.Q.A,.Q.n,"_-"; / chars allowed in a bare key; anything else must be "quoted" + +trimstr:{[s] i:where not s in " \t"; $[count i;(first i)_(1+last i)#s;""]}; / strip leading/trailing space+tab; keep internal +isquoted:{[s] (1 list of scalars (empty -> ()); else a single scalar. empty and unterminated + / values are rejected. + v:trimstr v; + if[0=count v;'"di.toml: missing value"]; + if["["=first v; + if[not "]"=last v;'"di.toml: malformed array (must be [ ... ]): ",v]; + inner:trimstr 1_-1_v; + :$[0=count inner;();parsescalar each splitcommas inner]]; + parsescalar v}; + +addkv:{[d;k;v] + / add k->v, signalling on a duplicate (a repeated key/section, or a key/section clash - TOML errors). + / catenation (not ,:) keeps the value list general so mixed value types never hit a type-widen error. + if[k in key d;'"di.toml: duplicate key: ",string k]; + d,(enlist k)!enlist v}; + +addline:{[acc;line] + / fold step; acc is (top;cursect;sect). a [section] flushes the open section into top and opens a + / fresh one; a key=value goes into the current section, or top before any section. + top:acc 0; cursect:acc 1; sect:acc 2; + $[(first line)="["; + [hdr:trimstr line; + if[not "]"=last hdr;'"di.toml: malformed section header (missing ]): ",line]; + if["[["~2 sublist hdr;'"di.toml: array-of-tables [[...]] is not supported: ",line]; + nm:sectname hdr; + if[not null cursect;top:addkv[top;cursect;sect]]; + (top;nm;()!())]; + [kv:splitassign line; + k:parsekey kv 0; + v:parsevalue kv 1; + $[null cursect;(addkv[top;k;v];cursect;sect);(top;cursect;addkv[sect;k;v])]]]}; + +parsetoml:{[text] + / TOML text -> dict (one level of [section] nesting). accepts a string (or the char atom q makes of + / a 1-char literal); other types signal - else a 1-char input is an atom and ssr/vs throw 'type. + if[not 10h=abs type text;'"di.toml: parsetoml expects a char string; got type ",string type text]; + lines:trimstr each stripcomment each "\n" vs $[10h=type text;text;enlist text]; + acc:addline/[(()!();`;()!());lines where 0value (sections nest)"); + (`parsefile; 1b; "read and parse a .toml file into a dict (signals if missing)"; "[string: file path]"; "dict: setting->value")); + }; From 40f803506794649e451dbbafb550ea7f9d41f702 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Wed, 5 Aug 2026 15:32:31 +0100 Subject: [PATCH 05/10] refactor to fit merged modules rather than mocked. Handlers module still mocked awaiting merge --- di/servers/servers.md | 28 +++++++++++++++++----------- di/servers/servers.q | 15 ++++++++++++--- di/servers/test.csv | 17 ++++++++++------- di/servers/test.q | 17 +++++++++-------- 4 files changed, 48 insertions(+), 29 deletions(-) diff --git a/di/servers/servers.md b/di/servers/servers.md index c4d893ce..df8c5203 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -23,8 +23,8 @@ duplicate `di.timer.addjob` id would throw). `init` does **not** open connection | key | kind | meaning | |---|---|---| -| `log` | injectable | binary `` `info`warn`error `` `{[c;m]}` logger dict | -| `timer` | injectable | di.timer contract; `addjob` = the 6-arg `custom` form `{[id;func;params;period;mode;opts]}` | +| `log` | injectable | binary `` `info`warn`error `` `{[c;m]}` logger dict — di.log's `logdict``log` satisfies this directly (it carries all six levels; the extra `trace`/`debug`/`fatal` are ignored) | +| `timer` | injectable | the di.timer export dict; di.servers calls `` timer[`addjob][`custom] `` — the 6-arg `{[id;func;params;period;mode;opts]}` variant (`addjob` is a variant dict of `custom`/`default`/`simple`) | | `handlers` | injectable | di.handlers contract; `register[event;phase;nm;pri;func]` | | `proctype`/`procname` | config | this process's own identity (required); used to exclude self from `process.csv` | | `connections` | config | proctypes this process should dial (symbols, or strings from a `.toml` cascade — normalised). Optional; default = none | @@ -100,17 +100,23 @@ priority-ordered fan-out. ## Open items / not yet done -- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 33 checks). They spawn a - genuinely separate `q` peer (a self-connect returns pseudo-handle `0`, not a real socket) and +- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 37 checks), and now wire the + **real merged `di.timer` and `di.log`** — only `di.handlers` (not yet merged) is mocked. They spawn + a genuinely separate `q` peer (a self-connect returns pseudo-handle `0`, not a real socket) and cover: `startup` connecting to a live peer and logging a failed dial while excluding self, `gethandlebytype` returning a live remote handle (`2=h"1+1"`), the retry cycle recovering an - ungraceful kill (`cleanup`+reopen), and `waitfortype` connected-vs-timeout — plus the mockable - surface (init validation, dep-wiring, idempotency, input validation, `getapimeta`). Because - `retry`/`cleanup` are internal, the retry cycle is driven by invoking the callback the **mock - timer captured** at `addjob` (the actually-wired path), not a direct export. -- **Provider modules not in kdbx-modules yet.** `di.log` (feature-logging branch) and - `di.handlers` aren't here yet, so the injected contracts are mocked in tests. The handlers - mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`. + ungraceful kill (`cleanup`+reopen), and `waitfortype` connected-vs-timeout — plus init validation, + dep-wiring, idempotency, input validation, and `getapimeta`. `init` schedules `serversretry` in the + real `di.timer` (asserted via `` timer.getalljobs[] ``); because `retry`/`cleanup` are internal, the + retry cycle is driven by invoking the exact func di.servers handed the timer (`` firejob `` reads it + back from `` getalljobs[] `` — the actually-wired path), not a direct export. Idempotent re-init is + a genuine test here: the real timer's `` addjob[`custom] `` throws on a duplicate id, so a + non-idempotent `init` would fail outright. A final check re-inits against di.log's real `logdict` to + prove the injected-log contract holds end-to-end. +- **`di.handlers` not in kdbx-modules yet**, so only that injected contract is mocked. The handlers + mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`; the `.z.pc` observer + path is therefore exercised only via the explicit `retry`→`cleanup` sweep, not a live auto-fired + `.z.pc` (which real di.handlers would install). - **`config`processcsv` and the assembled `connections` list** depend on di.torq's config wiring — coordinate when di.torq's servers dep is built. - Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and the diff --git a/di/servers/servers.q b/di/servers/servers.q index 150f830c..76051bdb 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -55,6 +55,12 @@ init:{[deps] '"di.servers: log dict must have `info`warn`error keys; got: ",(", " sv string key deps`log)]; if[99h<>type deps`timer; '"di.servers: timer value must be a dict (see di.timer)"]; + if[not `addjob in key deps`timer; + '"di.servers: timer dict must expose `addjob (see di.timer)"]; + if[99h<>type deps[`timer]`addjob; + '"di.servers: timer`addjob must be a variant dict (see di.timer addjob.custom/default/simple)"]; + if[not `custom in key deps[`timer]`addjob; + '"di.servers: timer`addjob must expose the `custom variant [id;func;params;period;mode;opts]"]; if[99h<>type deps`handlers; '"di.servers: handlers value must be a dict (see di.handlers)"]; if[not all `proctype`procname in key deps; @@ -76,9 +82,12 @@ init:{[deps] / (param `wh`, not `w`, so it does not shadow the SERVERS column w.) pcfunc:{[wh] .z.m.SERVERS:update endp:.z.p,w:0Ni from .z.m.SERVERS where w=wh; }; (.z.m.handlers[`register])[`.z.pc;`;`servers;0j;pcfunc]; - / di.timer mode-1h period is in SECONDS, so 10 = a 10-second retry (a bare 10000 here would be - / ~2.8h - the latent typo that made dead-handle recovery effectively never fire in early POCs). - (.z.m.timer[`addjob])[`serversretry;retry;();10;1;()!()]; + / di.timer's addjob is a VARIANT DICT; take `custom - the fully-configurable 6-arg form + / [id;func;params;period;mode;opts]. mode-1h period is in SECONDS, so 10 = a 10s retry (a bare + / 10000 would be ~2.8h - the latent typo that made dead-handle recovery never fire in early POCs). + / retry is passed BY VALUE (a lambda, not a symbol) so di.timer stores and runs it directly; its + / compile-time .z.m rewrite means it still updates di.servers' SERVERS when the timer fires it. + (.z.m.timer[`addjob][`custom])[`serversretry;retry;();10;1;()!()]; .z.m.registered:1b; ]; .z.m.loginfo[`init;"di.servers initialised"]; diff --git a/di/servers/test.csv b/di/servers/test.csv index 652476e8..3502d6ba 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -7,16 +7,16 @@ before,0,0,q,spawnpeer[],1,1,launch a genuinely separate q peer to act as otherp comment,,,,,,,init - dependency + config validation (plain signal; logger not wired yet) fail,0,0,q,svc.init[()!()],1,1,init with no deps errors fail,0,0,q,svc.init[enlist[`log]!enlist mocklog],1,1,init missing timer/handlers errors -fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(42;mocktimer;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a non-dict log value -fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!((enlist`info)!enlist{[c;m]};mocktimer;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a log dict missing warn/error -fail,0,0,q,"svc.init[`log`timer`handlers!(mocklog;mocktimer;mockhandlers)]",1,1,init rejects deps without proctype/procname identity +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(42;rtmr;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a non-dict log value +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!((enlist`info)!enlist{[c;m]};rtmr;mockhandlers;`selfproc;`selfinst)]",1,1,init rejects a log dict missing warn/error +fail,0,0,q,"svc.init[`log`timer`handlers!(mocklog;rtmr;mockhandlers)]",1,1,init rejects deps without proctype/procname identity comment,,,,,,,init - wiring and idempotency run,0,0,q,svc.init[svrdeps[`otherproc`deadproc]],1,1,init with mock deps + config (dial otherproc and deadproc) true,0,0,q,"1=count select from handlercalls where event=`.z.pc,name=`servers",1,1,init registered its .z.pc observer via the handlers dep -true,0,0,q,1=count select from timercalls where id=`serversretry,1,1,init scheduled the retry job via the timer dep -true,0,0,q,10=first exec period from timercalls where id=`serversretry,1,1,retry period is 10s (di.timer mode-1h period in seconds - guards the old 10000 typo) +true,0,0,q,1=count select from rtmr.getalljobs[] where id=`serversretry,1,1,init scheduled the retry job via the timer dep +true,0,0,q,10=first exec period from rtmr.getalljobs[] where id=`serversretry,1,1,retry period is 10s (di.timer mode-1h period in seconds - guards the old 10000 typo) run,0,0,q,svc.init[svrdeps[`otherproc`deadproc]],1,1,re-init must be idempotent -true,0,0,q,1=count select from timercalls where id=`serversretry,1,1,idempotent re-init did NOT add a second retry job +true,0,0,q,1=count select from rtmr.getalljobs[] where id=`serversretry,1,1,idempotent re-init did NOT add a second retry job true,0,0,q,"1=count select from handlercalls where event=`.z.pc,name=`servers",1,1,idempotent re-init did NOT re-register the pc handler comment,,,,,,,startup - connect to a live peer and a dead one (self excluded) run,0,0,q,svc.startup[],1,1,open connections to otherproc (live) and deadproc (nothing listening) @@ -42,8 +42,11 @@ comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout -fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;mocktimer;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype +fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;rtmr;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype true,0,0,q,(asc (key svc) except `init`getapimeta)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (plumbing omitted) true,0,0,q,not any `init`getapimeta in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta not registered) true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapimeta rows carry the registry columns +comment,,,,,,,real di.log integration (the merged logger itself - not the recording mock) +true,0,0,q,all `info`warn`error in key (use`di.log)[`logdict]`log,1,1,di.log logdict provides the info/warn/error keys the injected-log contract requires +run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!((use`di.log)[`logdict]`log;rtmr;mockhandlers;`selfproc;`selfinst)]",1,1,init accepts di.log's real logdict end-to-end (idempotent re-init against the merged logger) after,0,0,q,teardownfixture[],1,1,kill the peer and remove the fixture dir diff --git a/di/servers/test.q b/di/servers/test.q index 94474039..7de66186 100644 --- a/di/servers/test.q +++ b/di/servers/test.q @@ -10,12 +10,11 @@ mocklog:`info`warn`error!( {[c;m]`logrows upsert(`warn;c;m)}; {[c;m]`logrows upsert(`error;c;m)}); -/ timer mock: records (id;period) for the wiring asserts AND captures each job's func by id, so a -/ test can fire the retry cycle exactly as the real timer would (retry/cleanup are INTERNAL - not -/ exported - so they are driven only via this captured callback). -timercalls:([]id:`symbol$();period:`long$()); -timerjobs:(`symbol$())!(); -mocktimer:enlist[`addjob]!enlist {[id;func;params;period;mode;opts] timerjobs[id]:func; `timercalls upsert (id;period);}; +/ REAL di.timer - di.servers is its first consumer, so we integrate against the merged module (not a +/ mock's guess at the addjob contract) to prove the addjob[`custom] wiring end-to-end. di.servers does +/ NOT call timer.init, so no .z.ts cycle starts here: init adds the serversretry job to di.timer's jobs +/ table, and firejob invokes the stored func manually - exactly what the timer's cycle would do. +rtmr:use`di.timer; / handlers mock: records (event;name) with di.handlers' register[event;phase;nm;pri;func] shape. it / does NOT actually bind .z.pc - so the only cleanup path exercised here is the explicit retry-> @@ -27,7 +26,9 @@ mockhandlers:`register`remove`list!( {[ev]}); warnlogged:{[s] any (exec msg from logrows where lvl=`warn) like "*",s,"*"}; -firejob:{[id] timerjobs[id][]}; +/ fire a scheduled job's stored func exactly as the real di.timer's cycle would (retry/cleanup are +/ INTERNAL - not exported - so this is the only handle on them: the func di.servers gave the timer). +firejob:{[jid] (first exec func from rtmr.getalljobs[] where id=jid)[]}; / --- real peer process fixture --- FIXDIR:"/tmp/diserverstest"; @@ -67,4 +68,4 @@ setupfixture:{[] teardownfixture:{[] killpeer[]; system "rm -rf ",FIXDIR;}; / build the deps dict di.torq would assemble: injectables + this process's config slice. -svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;mocktimer;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; +svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; From cd427284d8e7153a8408995c156b179ce47edab0 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Thu, 6 Aug 2026 10:46:51 +0100 Subject: [PATCH 06/10] cleaning up orphaned code --- di/servers/servers.md | 13 +++++++------ di/servers/servers.q | 35 ++++++++++++++++++++--------------- di/servers/test.csv | 6 ++++++ di/servers/test.q | 4 ++++ 4 files changed, 37 insertions(+), 21 deletions(-) diff --git a/di/servers/servers.md b/di/servers/servers.md index df8c5203..9a793f78 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -43,7 +43,7 @@ h "1+1" | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | -| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). | | `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | @@ -88,9 +88,9 @@ priority-ordered fan-out. - **Three-flat-var logging** — `.z.m.loginfo`/`.z.m.logwarn`/`.z.m.logerr`, matching `consistency.md`, `di.compression` and `di.config`. (The project hasn't globally frozen this vs. the single-dict form — flag before changing.) -- **`raiseerror` (log-then-signal)** for all post-init domain errors (`formathp` unknown - ipctype, `selector` unknown selection, missing `process.csv`). `init`'s own dependency - validation is the one exception (plain `'` — no logger yet). +- **`raiseerror` (log-then-signal)** for all post-init domain errors (`selector` unknown + selection, missing or malformed `process.csv`). `init`'s own dependency validation is the one + exception (plain `'` — no logger yet). - **`getapimeta`** exported; a test asserts it documents exactly the module's *callable* exports — `init`/`getapimeta` are plumbing (di.torq calls them by convention) and are deliberately omitted from the registry rows, matching di.toml and the skill convention. No @@ -119,8 +119,9 @@ priority-ordered fan-out. `.z.pc` (which real di.handlers would install). - **`config`processcsv` and the assembled `connections` list** depend on di.torq's config wiring — coordinate when di.torq's servers dep is built. -- Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and the - `tcps`/`unix` socket types end-to-end (only `tcp` is wired through `startup`). +- Scoped-out (v1): discovery service, password/access-list files, non-TorQ tracking, and + `tcps`/`unix` socket types. Only `tcp` is supported; `formathp` builds a `tcp` handle with no + socket-type arg — a future `SOCKETTYPE` config reintroduces that (with a test) when needed. ## Tests diff --git a/di/servers/servers.q b/di/servers/servers.q index 76051bdb..c582f0eb 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -93,15 +93,11 @@ init:{[deps] .z.m.loginfo[`init;"di.servers initialised"]; }; -formathp:{[host;port;ipctype] - / internal - build a connection-handle symbol for `tcp`/`tcps`/`unix. only `tcp is exercised by - / startup in v1; the others exist for a future SOCKETTYPE-style config. - h:string host; - p:string port; - $[ipctype=`tcp; lower `$":",h,":",p; - ipctype=`tcps;lower `$":tcps://",h,":",p; - ipctype=`unix;lower `$":unix://",p; - raiseerror[`formathp;"unknown ipctype ",string ipctype]] +formathp:{[host;port] + / internal - build the tcp connection-handle symbol from a process.csv row. v1 is tcp only; a + / future SOCKETTYPE config would reintroduce tcps/unix handling (and a type arg) when there is a + / real requirement and a test - we do not ship unexercised branches. + lower `$":",(string host),":",string port }; opencon:{[hpup] @@ -114,12 +110,17 @@ opencon:{[hpup] }; readprocesscsv:{[path] - / internal - read the static process.csv phone book (host,port,proctype,procname). the PATH is - / supplied by the caller (from config`processcsv); di.servers reads no env itself, holding - / di.config's env-free boundary - di.torq resolves the path and puts it in config. + / internal - read the static process.csv phone book. the PATH comes from config`processcsv (di.torq + / resolves it; di.servers reads no env). v1 is a STRICT 4-column host,port,proctype,procname layout: + / validate the header up front and FAIL LOUD, because ("SISS";",") is positional and would otherwise + / silently misread a reordered or wider file (e.g. a real 13-column TorQ process.csv) into garbage. fsym:`$":",path; if[0=count key fsym;raiseerror[`readprocesscsv;"process.csv not found at ",path]]; - ("SISS";enlist",") 0: fsym + lines:read0 fsym; + if[0=count lines;raiseerror[`readprocesscsv;"process.csv is empty at ",path]]; + if[not `host`port`proctype`procname~`$trim each "," vs first lines; + raiseerror[`readprocesscsv;"process.csv header must be exactly host,port,proctype,procname (v1 4-column phone book); got: ",first lines]]; + ("SISS";enlist",") 0: lines }; startup:{[] @@ -139,9 +140,13 @@ startup:{[] procs:update isme:(proctype=pt)&procname=pn from procs; procs:select from procs where not isme; procs:select from procs where proctype in conns; - if[0=count procs;.z.m.loginfo[`servers;"no process.csv rows match the configured connections"];:()]; + / idempotent: skip any proc already tracked in SERVERS. a repeat startup (or a process.csv that has + / grown since) then adds only NEW rows - never a duplicate row or a leaked second handle to a proc + / already connected. reconnecting a dropped peer is retry's job, not startup's. + procs:select from procs where not procname in exec procname from .z.m.SERVERS; + if[0=count procs;.z.m.loginfo[`servers;"no new process.csv rows to connect"];:()]; {[row] - hpup:formathp[row`host;row`port;`tcp]; + hpup:formathp[row`host;row`port]; w:opencon[hpup]; if[not null w;.z.m.loginfo[`servers;"connected to ",(string row`proctype),"/",(string row`procname)," at ",string hpup]]; / catenate+reassign, NOT `tablename insert - a symbol-based insert into `.z.m.SERVERS` misses diff --git a/di/servers/test.csv b/di/servers/test.csv index 3502d6ba..950ecfda 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -38,6 +38,12 @@ true,0,0,q,1=count svc.getservers[`otherproc],1,1,retry reconnected to otherproc comment,,,,,,,waitfortype - connected vs timeout true,0,0,q,svc.waitfortype[`otherproc;2000;200],1,1,returns 1b immediately for an already-connected proctype true,0,0,q,0b~svc.waitfortype[`deadproc;700;200],1,1,returns 0b on timeout for a proctype that never comes up +comment,,,,,,,startup is idempotent - a repeat call must not duplicate rows or leak a second handle +run,0,0,q,svc.startup[],1,1,call startup a second time with the same config +true,0,0,q,1=count svc.getservers[`otherproc],1,1,repeat startup added no duplicate otherproc row (guard skips already-tracked procs) +comment,,,,,,,readprocesscsv fails loud on a reordered/wider header instead of silently misparsing +run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;`rdb;writebadcsv[])]",1,1,re-init pointing at a process.csv whose header is reordered +fail,0,0,q,svc.startup[],1,1,startup signals on the bad header rather than silently misreading columns comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection diff --git a/di/servers/test.q b/di/servers/test.q index 7de66186..47890bf8 100644 --- a/di/servers/test.q +++ b/di/servers/test.q @@ -67,5 +67,9 @@ setupfixture:{[] teardownfixture:{[] killpeer[]; system "rm -rf ",FIXDIR;}; +/ write a process.csv whose header is REORDERED vs the assumed host,port,proctype,procname (the exact +/ shape that used to silently misparse) and return its path - used to prove readprocesscsv fails loud. +writebadcsv:{[] (`$":",p:FIXDIR,"/bad.csv") 0: ("port,host,proctype,procname"; "5010,localhost,rdb,rdb1"); p}; + / build the deps dict di.torq would assemble: injectables + this process's config slice. svrdeps:{[conns] `log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;conns;FIXDIR,"/process.csv")}; From 395861f5c557783677d81b39023e8286583eefb3 Mon Sep 17 00:00:00 2001 From: ascottDI Date: Thu, 6 Aug 2026 15:24:26 +0100 Subject: [PATCH 07/10] adding in the version control --- di/servers/init.q | 8 +++++++- di/servers/servers.md | 17 ++++++++++++----- di/servers/test.csv | 8 ++++++-- di/servers/version.txt | 1 + 4 files changed, 26 insertions(+), 8 deletions(-) create mode 100644 di/servers/version.txt diff --git a/di/servers/init.q b/di/servers/init.q index 97ceb222..31fe018f 100644 --- a/di/servers/init.q +++ b/di/servers/init.q @@ -1,3 +1,9 @@ / connection management and handle-by-type lookup for the modular torq world. \l ::servers.q -export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta]) +/ module version: fallback default here, exported so di.depcheck can read di.servers' version to +/ satisfy other modules' declared minimums. +version:"0.1.0"; +/ version.txt in the module folder is the source of truth and takes priority over the fallback above; +/ read module-relative at load (`:::` resolves to di/servers). a missing/empty file keeps the fallback. +version:@[{trim first read0 x};`:::version.txt;{[e]version}]; +export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) diff --git a/di/servers/servers.md b/di/servers/servers.md index 9a793f78..a776a278 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -47,10 +47,11 @@ h "1+1" | `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | -| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta` plumbing omitted), for `di.torq` to register with `di.api`. | +| `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta`/`version` plumbing omitted), for `di.torq` to register with `di.api`. | +| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Sourced from `version.txt` (priority) with an `init.q` fallback; read by `di.depcheck` to satisfy other modules' declared minimum-version requirements. | Export is deliberately conservative — only functions `di.torq` or a consumer actually calls -(so `di.api` lists exactly these). The rest are **internal**: `retry` (the scheduled +(so `di.api` lists exactly these), plus the `version` metadata string. The rest are **internal**: `retry` (the scheduled `serversretry` job — passed to the timer *by value* at init, so it needs no export; it first runs `cleanup` to sweep ungracefully-vanished handles, then reopens every dead handle), `cleanup`, `formathp`, `opencon`, `readprocesscsv`, `retryrows`, `selector`, `updatestats`, @@ -92,9 +93,15 @@ priority-ordered fan-out. selection, missing or malformed `process.csv`). `init`'s own dependency validation is the one exception (plain `'` — no logger yet). - **`getapimeta`** exported; a test asserts it documents exactly the module's *callable* - exports — `init`/`getapimeta` are plumbing (di.torq calls them by convention) and are - deliberately omitted from the registry rows, matching di.toml and the skill convention. No - `version` export / VERSION file yet — deferred to the di.depcheck rollout, as in di.config. + exports — `init`/`getapimeta`/`version` are plumbing/metadata (di.torq calls or reads them by + convention) and are deliberately omitted from the registry rows, matching di.toml and the skill + convention. +- **`version` export** — a bare exported semver string (`"0.1.0"`, numeric `major.minor.patch`), + read by di.depcheck to satisfy other modules' declared minimum-version requirements. The source + of truth is **`version.txt`** in the module folder: `init.q` reads it module-relative at load + (`:::version.txt`) and it takes **priority** over the compiled-in fallback in `init.q`; a missing + or empty `version.txt` falls back to that default. Bump the release version by editing + `version.txt` alone. - **Env-free** — di.servers reads no environment variable; the `process.csv` path arrives via `config`processcsv` (di.torq resolves it), holding di.config's env-free boundary. diff --git a/di/servers/test.csv b/di/servers/test.csv index 950ecfda..45d1a2b4 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -49,9 +49,13 @@ fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proc fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;rtmr;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype -true,0,0,q,(asc (key svc) except `init`getapimeta)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (plumbing omitted) -true,0,0,q,not any `init`getapimeta in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta not registered) +true,0,0,q,(asc (key svc) except `init`getapimeta`version)~asc exec name from svc.getapimeta[],1,1,getapimeta documents exactly the callable exports (version/plumbing omitted) +true,0,0,q,not any `init`getapimeta`version in exec name from svc.getapimeta[],1,1,getapimeta omits plumbing (init/getapimeta/version not registered) true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapimeta rows carry the registry columns +comment,,,,,,,module metadata - exported version +true,0,0,q,10h=type svc.version,1,1,version is a string +true,0,0,q,0 Date: Tue, 11 Aug 2026 20:03:39 +0100 Subject: [PATCH 08/10] Update version convention, di.server name changed to di.servers, restore getservers ALL/null contract to couple correctly with di.heartbeat and prevent self-connections --- di/servers/{version.txt => VERSION} | 0 di/servers/init.q | 11 +++++------ di/servers/servers.md | 4 ++-- di/servers/servers.q | 25 +++++++++++++++++++++---- di/servers/test.csv | 16 +++++++++++++++- di/servers/test.q | 10 ++++++++++ 6 files changed, 53 insertions(+), 13 deletions(-) rename di/servers/{version.txt => VERSION} (100%) diff --git a/di/servers/version.txt b/di/servers/VERSION similarity index 100% rename from di/servers/version.txt rename to di/servers/VERSION diff --git a/di/servers/init.q b/di/servers/init.q index 31fe018f..2f6ca55b 100644 --- a/di/servers/init.q +++ b/di/servers/init.q @@ -1,9 +1,8 @@ / connection management and handle-by-type lookup for the modular torq world. \l ::servers.q -/ module version: fallback default here, exported so di.depcheck can read di.servers' version to -/ satisfy other modules' declared minimums. -version:"0.1.0"; -/ version.txt in the module folder is the source of truth and takes priority over the fallback above; -/ read module-relative at load (`:::` resolves to di/servers). a missing/empty file keeps the fallback. -version:@[{trim first read0 x};`:::version.txt;{[e]version}]; +/ module version, read from the VERSION file rather than hardcoded, so a release bump touches one +/ plain-text file. read module-relative at load (`:::` resolves to di/servers), and BEFORE the export +/ line since export:([...]) evaluates each name. NB `version` stays in the export: di.depcheck +/ resolves a dependency's version from the export dict +version:first read0`:::VERSION export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) diff --git a/di/servers/servers.md b/di/servers/servers.md index a776a278..5227fd5f 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -43,8 +43,8 @@ h "1+1" | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | -| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). | -| `getservers` | `getservers[proctype]` | Live (`w` non-null) `SERVERS` rows for a proctype. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). **Self-exclusion is an exact `(proctype;procname)` match against the identity from config**, so a `process.csv` that disagrees would leave this process dialling itself; any surviving row carrying this process's `procname` is therefore skipped with a warning naming both proctypes. | +| `getservers` | ``getservers[proctype]`` | Live (`w` non-null) `SERVERS` rows. Accepts a **symbol**, a **symbol list** (rows for any of them), or `` ` `` for **every** proctype — the contract legacy TorQ's `.servers.getservers` (`trackservers.q:75`) and sibling `di.serverselect.getservers` both implement, so a consumer written against either works here unchanged. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | | `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta`/`version` plumbing omitted), for `di.torq` to register with `di.api`. | diff --git a/di/servers/servers.q b/di/servers/servers.q index c582f0eb..92af0c5a 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -140,6 +140,17 @@ startup:{[] procs:update isme:(proctype=pt)&procname=pn from procs; procs:select from procs where not isme; procs:select from procs where proctype in conns; + / SELF-CONNECTION GUARD. the exclusion above is an exact (proctype;procname) match against the + / identity from config. if process.csv disagrees - a drifted proctype for this procname - the self + / row is not recognised and this process dials ITSELF. procname is process.csv's unique key (the + / idempotency filter below relies on that too), so a surviving row carrying our procname IS us: + / drop it and say why, rather than opening a self-connection nobody would think to look for. + / checked AFTER the connections filter: a row that could never be connected to was never at risk + if[count mismatched:select from procs where procname=pn; + .z.m.logwarn[`startup;"process.csv lists procname ",(string pn)," as proctype ", + (string first mismatched`proctype),", but this process is configured as proctype ",(string pt), + " - identity drift, skipping that row rather than connecting to myself"]; + procs:select from procs where not procname=pn]; / idempotent: skip any proc already tracked in SERVERS. a repeat startup (or a process.csv that has / grown since) then adds only NEW rows - never a duplicate row or a leaked second handle to a proc / already connected. reconnecting a dropped peer is retry's job, not startup's. @@ -183,9 +194,15 @@ retry:{[] }; getservers:{[pt] - / every live (non-null handle) SERVERS row for a proctype. - if[not -11h=type pt;raiseerror[`getservers;"proctype must be a symbol"]]; - select from .z.m.SERVERS where proctype=pt, not null w + / every live (non-null handle) SERVERS row for a proctype. ` matches EVERY proctype and a list + / matches any of them - the contract legacy TorQ's .servers.getservers (trackservers.q:75) and + / di.serverselect.getservers both implement, and which any consumer written against either expects. + / matching with `in` rather than `=` is what makes both shapes work; a bare symbol still behaves + / exactly as before, so every existing caller is unaffected + if[not 11h=abs type pt;raiseerror[`getservers;"proctype must be a symbol or symbol list"]]; + $[`~pt; + select from .z.m.SERVERS where not null w; + select from .z.m.SERVERS where proctype in pt, not null w] }; selector:{[tab;selection] @@ -246,7 +263,7 @@ getapimeta:{[] / listed - the registry describes the callable api, not plumbing. names are bare (di.torq qualifies). :flip `name`public`descrip`params`return!flip( (`startup; 1b; "open connections to configured proctypes from process.csv (reads init config)"; "[]"; "null"); - (`getservers; 1b; "live SERVERS rows for a proctype"; "[symbol: proctype]"; "table: live server rows"); + (`getservers; 1b; "live SERVERS rows for a proctype, a list of proctypes, or ` for all"; "[symbol|list: proctype, or ` for all]"; "table: live server rows"); (`gethandlebytype; 1b; "one live handle for a proctype via any/roundrobin/last selection"; "[symbol: proctype; symbol: selection]"; "int: handle, 0Ni if none"); (`waitfortype; 1b; "block until a proctype connects or timeout elapses"; "[symbol: proctype; long: timeoutms; long: pollms]"; "boolean: 1b connected, 0b timed out")); }; diff --git a/di/servers/test.csv b/di/servers/test.csv index 45d1a2b4..11f13c66 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -24,6 +24,13 @@ true,0,0,q,0=count svc.getservers[`selfproc],1,1,self is excluded even though it true,0,0,q,1=count svc.getservers[`otherproc],1,1,otherproc connected successfully true,0,0,q,0=count svc.getservers[`deadproc],1,1,deadproc failed (nothing listening) so it is not live true,0,0,q,"warnlogged[""failed to open connection""]",1,1,the failed dial was logged at warn +comment,,,,,,,"getservers contract - ` matches EVERY proctype and a list matches any of them (legacy trackservers.q:75 and di.serverselect both implement this; = against an atom was a regression against both)" +true,0,0,q,"(count svc.getservers[`])=sum count each svc.getservers each `selfproc`otherproc`deadproc",1,1,` returns exactly the union of the per-proctype results - match-all is not a special case with its own filter +true,0,0,q,1=count svc.getservers[`otherproc`deadproc],1,1,a list returns live rows for any listed proctype - deadproc is not live so contributes none +true,0,0,q,1=count svc.getservers[`otherproc`nosuchtype],1,1,an unknown proctype in the list is simply absent rather than an error +true,0,0,q,0=count svc.getservers[`nosuchtype`alsonone],1,1,a list matching nothing returns an empty table not an error +true,0,0,q,svc.getservers[`otherproc]~svc.getservers[enlist`otherproc],1,1,a one-element list is identical to the bare symbol - the existing atom path is unchanged +fail,0,0,q,svc.getservers[42],1,1,getservers still rejects a non-symbol non-list comment,,,,,,,gethandlebytype - a genuinely live remote handle run,0,0,q,h:svc.gethandlebytype[`otherproc;`any],1,1,get a handle to the peer true,0,0,q,"2=h ""1+1""",1,1,the handle is a live remote connection to a separate process @@ -46,6 +53,13 @@ run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname`connections`processcsv fail,0,0,q,svc.startup[],1,1,startup signals on the bad header rather than silently misreading columns comment,,,,,,,input validation + getapimeta fail,0,0,q,"svc.getservers[""nosuch""]",1,1,getservers rejects a non-symbol proctype + +comment,,,,,,,"startup must not dial ITSELF when process.csv disagrees with the configured identity" +run,0,0,q,"svc.init[`log`timer`handlers`proctype`procname`connections`processcsv!(mocklog;rtmr;mockhandlers;`selfproc;`selfinst;`rdb;writedriftcsv[])]",1,1,re-init against a csv listing OUR procname (selfinst) under a different proctype +run,0,0,q,delete from `logrows,1,1,clear the log capture +run,0,0,q,svc.startup[],1,1,startup - the exact (proctype;procname) match cannot recognise the drifted self row +true,0,0,q,0=count select from .m.di.0servers.SERVERS where procname=`selfinst,1,1,no row was added for this process - the guard skipped it rather than opening a connection to itself +true,0,0,q,"warnlogged[""connecting to myself""]",1,1,and the drift was reported rather than skipped silently fail,0,0,q,"svc.gethandlebytype[`hdb;""any""]",1,1,gethandlebytype rejects a non-symbol selection fail,0,0,q,"svc.waitfortype[`hdb;""x"";200]",1,1,waitfortype rejects a non-integer timeout fail,0,0,q,"svc.init[`log`timer`handlers`proctype`procname!(mocklog;rtmr;mockhandlers;""rdb"";`p)]",1,1,init rejects a non-symbol proctype @@ -55,7 +69,7 @@ true,0,0,q,`name`public`descrip`params`return~cols svc.getapimeta[],1,1,getapime comment,,,,,,,module metadata - exported version true,0,0,q,10h=type svc.version,1,1,version is a string true,0,0,q,0 Date: Thu, 13 Aug 2026 16:10:38 +0100 Subject: [PATCH 09/10] following automated reviewer comments --- di/servers/init.q | 8 ++++++-- di/servers/servers.md | 4 ++-- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/di/servers/init.q b/di/servers/init.q index 2f6ca55b..df5fd89c 100644 --- a/di/servers/init.q +++ b/di/servers/init.q @@ -3,6 +3,10 @@ / module version, read from the VERSION file rather than hardcoded, so a release bump touches one / plain-text file. read module-relative at load (`:::` resolves to di/servers), and BEFORE the export / line since export:([...]) evaluates each name. NB `version` stays in the export: di.depcheck -/ resolves a dependency's version from the export dict -version:first read0`:::VERSION +/ resolves a dependency's version from the export dict. +/ trim so a trailing newline/CRLF cannot pad the semver; fail loud with a clear message if VERSION is +/ missing/unreadable/empty (it is a required module file - better than a raw OS error, a silent empty +/ value, or a misleading 0.0.0 that would corrupt depcheck's version comparison). +version:@[{trim first read0 x};`:::VERSION;{'"di.servers: VERSION file missing or unreadable"}]; +if[0=count version;'"di.servers: VERSION file is empty"]; export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) diff --git a/di/servers/servers.md b/di/servers/servers.md index 5227fd5f..d008304e 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -43,12 +43,12 @@ h "1+1" | Function | Signature | Description | |---|---|---| | `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | -| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). **Self-exclusion is an exact `(proctype;procname)` match against the identity from config**, so a `process.csv` that disagrees would leave this process dialling itself; any surviving row carrying this process's `procname` is therefore skipped with a warning naming both proctypes. | +| `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). **Self-exclusion is an exact `(proctype;procname)` match against the identity from config**, so a `process.csv` that disagrees would leave this process dialling itself; any row carrying this process's `procname` **that survives the connections filter (i.e. one this process would otherwise dial)** is therefore skipped with a warning naming both proctypes. A drifted row whose proctype isn't a connection type is already filtered out and poses no self-connection risk, so it is not warned about. | | `getservers` | ``getservers[proctype]`` | Live (`w` non-null) `SERVERS` rows. Accepts a **symbol**, a **symbol list** (rows for any of them), or `` ` `` for **every** proctype — the contract legacy TorQ's `.servers.getservers` (`trackservers.q:75`) and sibling `di.serverselect.getservers` both implement, so a consumer written against either works here unchanged. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | | `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | | `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta`/`version` plumbing omitted), for `di.torq` to register with `di.api`. | -| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Sourced from `version.txt` (priority) with an `init.q` fallback; read by `di.depcheck` to satisfy other modules' declared minimum-version requirements. | +| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Read at load from the plain-text `VERSION` file in the module folder; read by `di.depcheck` to satisfy other modules' declared minimum-version requirements. | Export is deliberately conservative — only functions `di.torq` or a consumer actually calls (so `di.api` lists exactly these), plus the `version` metadata string. The rest are **internal**: `retry` (the scheduled From 46af90e1021410923a021c66234ca3535f2d2e0e Mon Sep 17 00:00:00 2001 From: alowrydi Date: Wed, 19 Aug 2026 15:44:04 +0100 Subject: [PATCH 10/10] di.servers: add requireinit guard and teardown, and validate the timer/handler deps at init Every exported function except getapimeta now refuses to run before init has completed. Previously a pre-init call failed silently rather than loudly: getservers returned an empty table, gethandlebytype 0Ni, and waitfortype spun its full timeout before returning 0b - all three indistinguishable from "genuinely nothing of that type is connected". Only startup threw, and then with a bare unnamed error off the unset connections config. The probe reads the injected logger, which has no load-time default; self and registered are both declared at load and so cannot tell "init ran" from "module loaded". teardown releases the two process-global registrations init installs - the .z.pc handler and the serversretry timer job - and resets the registered flag so a later init re-registers rather than being skipped by the idempotency guard. Module state, SERVERS included, is deliberately left intact so a shutdown path can still inspect what was connected. Idempotent: di.timer.deletejobs is a delete-where and di.handlers' removesimple early-returns, so a second call cannot die on what the first removed. init now checks that the timer dep exposes a callable deletejobs, and the handlers dep a callable register and remove. A dict hands back a null for a missing key, so an unvalidated dep turns teardown's @[f;ids;handler] from protected apply into three-argument amend: it upserts into a throwaway dict, deletes nothing, warns nothing, and teardown still reports success. The callable test excludes 101h (::) as well as testing the 100 112h range, because :: sits inside that range while being callable in no useful sense and is exactly what a function-valued dep dict returns for a missing key. Verified end to end - without the exclusion a null deletejobs passed init and left serversretry scheduled after teardown. Test fixtures resolve from the module directory via .Q.m.mp rather than the current working directory, so the suite runs from outside the repo root. Suite is 93 checks, all passing. Each new guard and check was confirmed by deletion: removing any one of them fails its own rows and no others. --- di/servers/init.q | 2 +- di/servers/servers.md | 165 ++++++++++++++++++++++++++++++++++++++---- di/servers/servers.q | 104 +++++++++++++++++++++++++- di/servers/test.csv | 75 ++++++++++++++++++- di/servers/test.q | 4 +- 5 files changed, 330 insertions(+), 20 deletions(-) diff --git a/di/servers/init.q b/di/servers/init.q index df5fd89c..ad8a274c 100644 --- a/di/servers/init.q +++ b/di/servers/init.q @@ -9,4 +9,4 @@ / value, or a misleading 0.0.0 that would corrupt depcheck's version comparison). version:@[{trim first read0 x};`:::VERSION;{'"di.servers: VERSION file missing or unreadable"}]; if[0=count version;'"di.servers: VERSION file is empty"]; -export:([init;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) +export:([init;teardown;startup;getservers;gethandlebytype;waitfortype;getapimeta;version]) diff --git a/di/servers/servers.md b/di/servers/servers.md index d008304e..dda421ea 100644 --- a/di/servers/servers.md +++ b/di/servers/servers.md @@ -17,15 +17,17 @@ dependencies **and** the config keys. `init` wires the deps, records self-identi installs two one-time process-global side effects — a `.z.pc` cleanup handler and a 10s retry timer job. It is **idempotent** (guarded by an internal `registered` flag): `di.torq` calls it once per process, but a second call refreshes the dep refs without re-registering (a -duplicate `di.timer.addjob` id would throw). `init` does **not** open connections. +duplicate `di.timer.addjob` id would throw). `teardown` resets that flag, so an +`init`→`teardown`→`init` cycle re-installs both side effects rather than being skipped by the +guard. `init` does **not** open connections. `deps` keys: | key | kind | meaning | |---|---|---| | `log` | injectable | binary `` `info`warn`error `` `{[c;m]}` logger dict — di.log's `logdict``log` satisfies this directly (it carries all six levels; the extra `trace`/`debug`/`fatal` are ignored) | -| `timer` | injectable | the di.timer export dict; di.servers calls `` timer[`addjob][`custom] `` — the 6-arg `{[id;func;params;period;mode;opts]}` variant (`addjob` is a variant dict of `custom`/`default`/`simple`) | -| `handlers` | injectable | di.handlers contract; `register[event;phase;nm;pri;func]` | +| `timer` | injectable | the di.timer export dict. Must expose `` `addjob `` (a variant dict of `custom`/`default`/`simple`; di.servers calls the 6-arg `` timer[`addjob][`custom] ``) **and** `` `deletejobs ``, which must be **callable** — `teardown` needs it, and a timer dep missing it fails *silently* there rather than loudly at `init` (see [Lifecycle](#lifecycle-requireinit-and-teardown)) | +| `handlers` | injectable | di.handlers contract. Must expose callable `` `register `` (`register[event;phase;nm;pri;func]` — `init` uses it) **and** `` `remove `` (`remove[event;phase;nm]` — `teardown` uses it) | | `proctype`/`procname` | config | this process's own identity (required); used to exclude self from `process.csv` | | `connections` | config | proctypes this process should dial (symbols, or strings from a `.toml` cascade — normalised). Optional; default = none | | `processcsv` | config | **path** to `process.csv`; supplied by di.torq. Optional; required only once `connections` is non-empty | @@ -42,21 +44,136 @@ h "1+1" | Function | Signature | Description | |---|---|---| -| `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. | +| `init` | `init[deps]` | Wire deps + config, record identity, install the `.z.pc` handler + retry job. Idempotent. **The only function that may be called before `init`** — every other one below (except `getapimeta`) refuses to run until it has completed. | +| `teardown` | `teardown[]` | Release both process-global registrations `init` installed — the `.z.pc` handler and the `serversretry` timer job. Module state, `SERVERS` included, is **deliberately left intact**. Idempotent. See [Lifecycle](#lifecycle-requireinit-and-teardown). | | `startup` | `startup[]` | Read `process.csv` (`processcsv`), drop self, connect to each row whose proctype is in `connections`. A failed connection is logged (not raised) and left as `w:0Ni` for `retry`. No-op if no connections configured. **Idempotent**: skips procs already tracked in `SERVERS`, so a repeat call (or a grown `process.csv`) adds only new rows — never a duplicate or a leaked second handle. `process.csv` must be the strict v1 4-column `host,port,proctype,procname` layout — a reordered or wider header is **rejected loudly** (the reader is positional, so it would otherwise misparse silently). **Self-exclusion is an exact `(proctype;procname)` match against the identity from config**, so a `process.csv` that disagrees would leave this process dialling itself; any row carrying this process's `procname` **that survives the connections filter (i.e. one this process would otherwise dial)** is therefore skipped with a warning naming both proctypes. A drifted row whose proctype isn't a connection type is already filtered out and poses no self-connection risk, so it is not warned about. | | `getservers` | ``getservers[proctype]`` | Live (`w` non-null) `SERVERS` rows. Accepts a **symbol**, a **symbol list** (rows for any of them), or `` ` `` for **every** proctype — the contract legacy TorQ's `.servers.getservers` (`trackservers.q:75`) and sibling `di.serverselect.getservers` both implement, so a consumer written against either works here unchanged. | | `gethandlebytype` | `gethandlebytype[proctype;selection]` | One live handle via `` `any``/`roundrobin`/`last``; `0Ni` if none. Bumps usage stats. | -| `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. | +| `waitfortype` | `waitfortype[proctype;timeoutms;pollms]` | Block until a live connection exists or timeout; `1b`/`0b`. Caller decides if a timeout is fatal. `startup` must have run first. A **zero or already-expired `timeoutms`** performs no active retry — see [waitfortype with a zero timeout](#waitfortype-with-a-zero-timeout). | | `getapimeta` | `getapimeta[]` | This module's api metadata, one row per **callable** API function (`init`/`getapimeta`/`version` plumbing omitted), for `di.torq` to register with `di.api`. | -| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Read at load from the plain-text `VERSION` file in the module folder; read by `di.depcheck` to satisfy other modules' declared minimum-version requirements. | +| `version` | `version` | The module's semver string (`"0.1.0"`) — metadata, not a function. Read at load from the plain-text `VERSION` file in the module folder; `di.depcheck` resolves it from the export dict to satisfy other modules' declared minimum-version requirements. | Export is deliberately conservative — only functions `di.torq` or a consumer actually calls (so `di.api` lists exactly these), plus the `version` metadata string. The rest are **internal**: `retry` (the scheduled `serversretry` job — passed to the timer *by value* at init, so it needs no export; it first runs `cleanup` to sweep ungracefully-vanished handles, then reopens every dead handle), `cleanup`, `formathp`, `opencon`, `readprocesscsv`, `retryrows`, `selector`, `updatestats`, -`signalfound`, `raiseerror`; plus state (`SERVERS`, `self`, `registered`, `HOPENTIMEOUT`, -`connections`, `processcsv`). +`signalfound`, `raiseerror`, `initialised`, `requireinit`, `iscallable`; plus state (`SERVERS`, `self`, +`registered`, `HOPENTIMEOUT`, `connections`, `processcsv`). + +## Lifecycle: requireinit and teardown + +### `requireinit` — nothing runs before `init` + +`startup`, `getservers`, `gethandlebytype`, `waitfortype` and `teardown` each call +`requireinit[ctx]` as their first statement and signal +`di.servers: : init must be called before any other function` if `init` has not completed. +`getapimeta` deliberately does **not** — it returns static metadata derived from no init'd state, +matching every other module in this project. + +This is not a nicety. `SERVERS`, `self`, `registered` and `HOPENTIMEOUT` are **load-time** +constants, so before the guard existed a pre-init call was *silently* wrong rather than loud: + +| call | before `init`, without the guard | indistinguishable from | +|---|---|---| +| `getservers[pt]` | an empty table | no process of that type is connected | +| `gethandlebytype[pt;sel]` | `0Ni` | nothing of that type is connected right now | +| `waitfortype[pt;t;p]` | spins the **full** timeout, then `0b` | a peer that never came up | + +Only `startup` threw at all, and then with a bare unnamed error off the unset `.z.m.connections`. +With no `di.torq` yet, every process here is wired by a hand-written script calling `init` on +several modules in order — exactly the situation where an ordering mistake is plausible — and a +failure that looks precisely like "the dependency is still starting up" is the worst shape this bug +could take. `di.rdb` already wraps its `gethandlebytype` calls protectively *on the assumption this +guard exists* (`rdb.q:656`). + +**Why the probe reads `.z.m.loginfo`.** `initialised[]` must read something whose value can only +come from `init`. `self` and `registered` cannot serve — both are declared at **load** time, so +neither distinguishes "init ran" from "the module was merely loaded". `.z.m.loginfo` has no +load-time default and is written only by `init`; it is also the truest probe of what the guard +protects, since without a wired logger the module cannot even report its own failures. + +It is deliberately **not** `registered`. `teardown` resets that flag, and conflating the two would +make every function report *"init must be called before any other function"* after a teardown — +untrue, and it would put `SERVERS` out of reach of exactly the shutdown path `teardown` leaves it +intact for. The two flags stay separate, as in `di.rdb` (probes `hdbdir`, resets `started`) and +`di.subscriptions` (probes `subscriptions`, resets `observing`). + +### `teardown[]` — what it releases, and what it does not + +`init` installs **two** process-global things, so there is something to give back: + +| released by `teardown` | how | +|---|---| +| the `.z.pc` cleanup handler | `` handlers[`remove][`.z.pc;`;`servers] `` — same event, phase and registrant name it was registered under | +| the `serversretry` timer job | `` timer[`deletejobs][`serversretry] `` | + +**Not** released — module state is deliberately left intact, the same convention every other +teardown in this project follows: + +- `SERVERS` keeps every row, so a shutdown path can still see what was connected and to whom. +- The logger, timer and handler dep refs, self-identity, `connections` and `processcsv` all stay. +- `getservers`, `gethandlebytype`, `waitfortype` and `startup` all remain callable. What is + withdrawn is only the *automatic* behaviour: with the `serversretry` job gone nothing reattempts + a dropped connection on a cycle, and with the `.z.pc` handler gone a clean disconnect is no longer + swept. `waitfortype` still drives `retry` itself, so it keeps working. There is no + `requireobserver`-style refusal on `startup` after a teardown. + +`teardown` resets `registered` to `0b`, so a later `init` installs both side effects again rather +than being skipped by the idempotency guard — a `teardown`→`init` cycle fully works. + +**Idempotent — a second `teardown[]` does not throw.** Both release calls are no-ops on an +already-removed registration, verified in the dependencies rather than assumed: +`di.timer.deletejobs` is a delete-where over its jobs table (`di/timer/init.q:95`), so an id that +is not there matches nothing; `di.handlers`' `removesimple` early-returns with an info log when the +event or the name is not registered (`di/handlers/handlers.q:71-72`). + +**Why `init` validates the timer's `deletejobs`.** A timer dep's value side is dict-typed, so a +*missing* key returns a null-shaped **dict** rather than erroring. `@[x;y;z]` is "try `x[y]`, catch +with `z`" only when `x` is a **function** — with a dict, `teardown`'s +`` @[.z.m.timer[`deletejobs];`serversretry;handler] `` is read as three-argument **amend** instead: +it upserts the id into that throwaway dict using the error handler as the value, discards the result +and carries on. Nothing throws, nothing warns, the job is never deleted, and `teardown` still logs +success — a false positive. Presence alone is not enough either: a non-callable `deletejobs` lands +in the identical amend. `init` therefore checks both presence and callable type, and the same pair +is applied to `handlers`' `register` and `remove`. The checks must be at `init` time because +`teardown` cannot detect the problem at all. + +The callable test is the internal `iscallable`, **not** a bare `` within 100 112h ``. That range +spans every genuinely callable form — lambda, primitive, operator, iterator, projection, +composition — but it also admits **`101h`, the generic null `::`**, which is callable in no useful +sense and is precisely what a dict hands back for a *missing* key when its value side is plain +functions (`` `deletejobs _ di.timer `` yields `101h`; only a *table*-valued dep yields the `99h` +null-shaped dict). Left in, a dep that was absent or explicitly null passed `init`, and `teardown` +then ran `` @[::;`serversretry;handler] `` — which simply returns the id, deleting nothing, while +`teardown` logged success. Measured end to end, which is why `iscallable` excludes `101h`: + +```q +iscallable:{[x] t:type x; :(t within 100 112h) and 101h<>t; }; +``` + +> **Known divergence.** `di.rdb` performs the equivalent check with a bare `` within 100 112h `` and +> therefore still carries this hole. It was left untouched deliberately — it is a separate module and +> its own unit of work — but the same tightening applies there. + +## `waitfortype` with a zero timeout + +**Decision: `waitfortype[pt;0;pollms]` performs no active retry, and that is intended.** With a +zero (or already-expired) `timeoutms` the deadline is in the past by the time the loop condition is +first evaluated, so `retry[]` is never called; the function reports current state, logs the usual +timeout warning, and returns `0b`. + +Reasoning, recorded rather than left implicit: + +- `retry[]` calls `opencon` for every dead row at `HOPENTIMEOUT` = 2000 ms. Granting a "0 ms" call + one active attempt could therefore block for **seconds** per unreachable peer — violating the + caller's explicit budget in the most surprising possible direction. A caller who asks for no wait + should get no wait. +- Nothing is lost. The scheduled `serversretry` job reattempts every 10 s regardless, so a reconnect + is *deferred*, not skipped. +- A caller who does want one active attempt asks for one by passing a `timeoutms` of at least + `pollms`; there is no need for a zero timeout to mean something different from what it says. +- The `0b` return still carries the existing timeout warning, so the outcome stays diagnosable. ## The `SERVERS` table @@ -82,6 +199,13 @@ must be `` ` `` (null) — di.handlers rejects a non-null phase on an observer e di.servers' disconnect hook coexist with every other `.z.pc` registrant in the same priority-ordered fan-out. +`teardown` gives the registration back through the same dependency, under the same event, phase +and registrant name: + +```q +(handlers[`remove])[`.z.pc;`;`servers] +``` + ## Conventions (learnings from di.config) - **One-arg `init[deps]`** with config folded into `deps` (the project convention; matches @@ -97,17 +221,18 @@ priority-ordered fan-out. convention) and are deliberately omitted from the registry rows, matching di.toml and the skill convention. - **`version` export** — a bare exported semver string (`"0.1.0"`, numeric `major.minor.patch`), - read by di.depcheck to satisfy other modules' declared minimum-version requirements. The source - of truth is **`version.txt`** in the module folder: `init.q` reads it module-relative at load - (`:::version.txt`) and it takes **priority** over the compiled-in fallback in `init.q`; a missing - or empty `version.txt` falls back to that default. Bump the release version by editing - `version.txt` alone. + read by di.depcheck to satisfy other modules' declared minimum-version requirements. The single + source of truth is the **`VERSION`** file in the module folder, read module-relative at load by + `init.q` and `trim`med so a trailing newline cannot pad the semver. There is no compiled-in + fallback: a missing, unreadable or empty `VERSION` signals a named error at load rather than + yielding a silently stale or `0.0.0` version that would corrupt di.depcheck's comparison. Bump the + release by editing `VERSION` alone. - **Env-free** — di.servers reads no environment variable; the `process.csv` path arrives via `config`processcsv` (di.torq resolves it), holding di.config's env-free boundary. ## Open items / not yet done -- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 37 checks), and now wire the +- **Live-peer integration tests are in place** (`test.q` + `test.csv`, 84 checks), and now wire the **real merged `di.timer` and `di.log`** — only `di.handlers` (not yet merged) is mocked. They spawn a genuinely separate `q` peer (a self-connect returns pseudo-handle `0`, not a real socket) and cover: `startup` connecting to a live peer and logging a failed dial while excluding self, @@ -120,6 +245,18 @@ priority-ordered fan-out. a genuine test here: the real timer's `` addjob[`custom] `` throws on a duplicate id, so a non-idempotent `init` would fail outright. A final check re-inits against di.log's real `logdict` to prove the injected-log contract holds end-to-end. + The lifecycle rows added alongside `requireinit`/`teardown` are worth knowing about when editing + the suite: the pre-init guard rows must stay **above every `init` attempt**, failing ones included + — an `init` validation gap would let one of those succeed, wire the logger, and make the guard rows + fail for that reason instead of a missing guard (measured). They assert the *named* message via + `ss`, not merely that something threw, since a bare `fail` row would pass on any unrelated throw + and a "returned empty" row would pass under the old silent behaviour. Each guard also asserts the + **function name** in the message: `gethandlebytype` calls `getservers` internally, so dropping its + own `requireinit` still produces an "init must be called" error — from the nested guard — and only + the function-name assertion catches it. `teardown` is covered for both its releases, a second call, + and an `init` after it re-installing both side effects (which is what proves the `registered` + reset). The handlers mock's `remove` deletes its recorded row rather than being a no-op, so a + broken `teardown` cannot pass. - **`di.handlers` not in kdbx-modules yet**, so only that injected contract is mocked. The handlers mock uses the real `register[event;phase;nm;pri;func]` shape from `handlers.q`; the `.z.pc` observer path is therefore exercised only via the explicit `retry`→`cleanup` sweep, not a live auto-fired diff --git a/di/servers/servers.q b/di/servers/servers.q index 92af0c5a..c3e52659 100644 --- a/di/servers/servers.q +++ b/di/servers/servers.q @@ -28,9 +28,49 @@ self:`proctype`procname!``; / guards init's one-time process-global side effects (the .z.pc observer + the retry timer job) so / init is IDEMPOTENT - di.torq calls it once per process, but a second call (a test re-run, a / future re-init) must not re-register: di.timer.addjob throws on a duplicate id. the dep refs are -/ always refreshed; only the one-time registrations are guarded. +/ always refreshed; only the one-time registrations are guarded. teardown resets it to 0b so a +/ later init registers again - it is the SIDE-EFFECT guard only, deliberately not the initialised[] +/ probe below (see there for why the two are separate flags). registered:0b; +initialised:{[] + / has init run? .z.m.loginfo has no load-time default - its only value comes from init, so this + / probe cannot be fooled by a constant of the same name. that rules out the two obvious + / alternatives: `self` and `registered` are both declared at LOAD time above, so neither can tell + / "init ran" from "the module was merely loaded". it is also the truest probe of what requireinit + / actually protects - without a wired logger the module cannot even report its own failures. + / deliberately NOT `registered`: teardown resets that, and conflating the two would make every + / function report "init must be called" after a teardown, which is both untrue and would put + / SERVERS out of reach of the shutdown path teardown exists to serve. di.rdb (probes hdbdir, + / resets started) and di.subscriptions (probes subscriptions, resets observing) split them the + / same way. + :@[{.z.m.loginfo;1b};::;{[e] :0b}]; + }; + +requireinit:{[ctx] + / every exported function except init/getapimeta refuses to run before init has wired the deps. + / without this a pre-init call is SILENTLY wrong rather than loud, which is the worst shape a bug + / here could take: getservers returns an empty table, gethandlebytype 0Ni, and waitfortype spins + / its full timeout before returning 0b - all three indistinguishable from "genuinely nothing of + / that type is connected right now". signals with a plain ' and NOT raiseerror: the logger is the + / very thing that may not be wired yet. + if[not initialised[]; + '"di.servers: ",string[ctx],": init must be called before any other function"]; + }; + +iscallable:{[x] + / internal - is x a genuinely callable value? 100 112h spans every callable form (lambda, + / primitive, operator, iterator, projection, composition), but 101h - the generic null :: - sits + / INSIDE that range while being callable in no useful sense, so the bare range check is not enough. + / that matters because :: is exactly what a dict hands back for a MISSING key when its value side + / is plain functions, which is what a real dep dict looks like (`deletejobs _ di.timer gives 101h, + / not the 99h a table-valued dep gives). admitting it lets an absent or explicitly-null dep reach + / teardown and no-op in silence: @[::;id;handler] simply returns id, deleting nothing, while + / teardown logs success. measured end to end, hence the extra exclusion + t:type x; + :(t within 100 112h) and 101h<>t; + }; + raiseerror:{[ctx;msg] / internal - log an error under ctx via the injected logger, then signal it, so a failure is / observable in the log as well as thrown. used for all post-init domain errors (init's own @@ -61,8 +101,36 @@ init:{[deps] '"di.servers: timer`addjob must be a variant dict (see di.timer addjob.custom/default/simple)"]; if[not `custom in key deps[`timer]`addjob; '"di.servers: timer`addjob must expose the `custom variant [id;func;params;period;mode;opts]"]; + / deletejobs is checked here for the same reason addjob is, and the consequence of NOT checking it + / is worse than a late error - it is a SILENT one. the timer dep's value side is dict-typed, so a + / MISSING key returns a null-shaped DICT rather than erroring. teardown's + / @[.z.m.timer[`deletejobs];ids;handler] then stops being protected-apply at all: @[x;y;z] is + / "try x[y], catch with z" only when x is a FUNCTION, and here x is a dict, so q reads the whole + / expression as three-argument AMEND. it upserts the job id into that throwaway dict using the + / error handler as the new value, discards the result (nothing in teardown captures it) and + / carries on. nothing throws, nothing warns, serversretry is never deleted, and teardown still + / logs success - a false positive. see di.rdb, which closes the identical trap + if[not `deletejobs in key deps`timer; + '"di.servers: timer dict must expose `deletejobs - teardown needs it, and a timer dep without ", + "it fails SILENTLY at teardown rather than loudly here; see di.timer"]; + / presence is NOT enough: a non-callable deletejobs reaches teardown's @[...] and lands in exactly + / the same amend interpretation as a missing key, so it too returns quietly having deleted + / nothing. iscallable, not a bare `within 100 112h`: the range admits 101h (::), which is both a + / non-callable and the exact value a function-valued dep dict returns for a missing key - see there + if[not iscallable deps[`timer]`deletejobs; + '"di.servers: timer`deletejobs must be a function [ids]; a non-callable or null value fails ", + "silently at teardown - see di.timer"]; if[99h<>type deps`handlers; '"di.servers: handlers value must be a dict (see di.handlers)"]; + / register is what init calls and remove is what teardown calls; neither was validated before, so + / a handlers dep missing either failed late and obscurely at the call site instead of here, naming + / neither the module nor the missing key. same presence-then-callable pair as the timer checks + if[not all `register`remove in key deps`handlers; + '"di.servers: handlers dict must have `register`remove keys (init registers, teardown removes); ", + "got: ",(", " sv string key deps`handlers)]; + if[not all iscallable each deps[`handlers]`register`remove; + '"di.servers: handlers`register and handlers`remove must both be functions, neither null ", + "- see di.handlers"]; if[not all `proctype`procname in key deps; '"di.servers: proctype and procname (self-identity) are required in deps"]; if[not all -11h=type each deps`proctype`procname; @@ -93,6 +161,34 @@ init:{[deps] .z.m.loginfo[`init;"di.servers initialised"]; }; +teardown:{[] + / release both process-global registrations init installed - the .z.pc observer and the + / serversretry timer job - so nothing of di.servers is left bound process-wide. paired with init's + / side effects, exactly as di.subscriptions.teardown is paired with its .z.pc registration and + / di.rdb.teardown with its root entry points and timer jobs. + / module state is deliberately LEFT INTACT - SERVERS above all - so a shutdown path can still + / inspect what was connected and to whom; only the process-global bindings are withdrawn. the same + / convention every other teardown in this project follows. + / IDEMPOTENT: a second call must not die on what the first already removed. both release calls are + / no-ops on an already-removed registration - di.timer.deletejobs is a delete-where over its jobs + / table (an id that is not there matches nothing) and di.handlers' removesimple early-returns with + / an info log when the event or the name is not registered. verified in both modules, not assumed. + requireinit[`teardown]; + / called directly rather than through @[...]: init guarantees `remove is present and callable, so + / a throw here is a genuine handlers failure and should surface rather than be swallowed + .z.m.handlers[`remove][`.z.pc;`;`servers]; + / NB this @[...] IS protected-apply only because init guarantees `deletejobs is present and is a + / function. were the key missing, x would be the null-shaped dict a dict-valued dep returns for an + / absent key and q would read the line as three-argument amend instead, deleting nothing and + / logging nothing while teardown reported success. the init check is what keeps this line honest + @[.z.m.timer[`deletejobs];`serversretry; + {[e] .z.m.logwarn[`teardown;"could not delete the serversretry timer job: ",e]}]; + / reset the SIDE-EFFECT guard only, so a later init registers both again. initialised[] probes + / .z.m.loginfo, not this, so everything else stays callable after a teardown - see initialised[] + .z.m.registered:0b; + .z.m.loginfo[`teardown;"di.servers .z.pc registration and serversretry job removed"]; + }; + formathp:{[host;port] / internal - build the tcp connection-handle symbol from a process.csv row. v1 is tcp only; a / future SOCKETTYPE config would reintroduce tcps/unix handling (and a type arg) when there is a @@ -130,6 +226,7 @@ startup:{[] / normalise connections to symbols to match process.csv's `proctype column (always a symbol via / the "S" spec): a .q settings file gives symbols already (`$ throws 'type on a symbol - it is / NOT idempotent, hence the type check); a .toml one gives plain strings (TOML has no symbol). + requireinit[`startup]; conns:.z.m.connections; conns:$[11h=abs type conns;conns;`$conns]; if[0=count conns;.z.m.loginfo[`servers;"no configured connections to make"];:()]; @@ -199,6 +296,7 @@ getservers:{[pt] / di.serverselect.getservers both implement, and which any consumer written against either expects. / matching with `in` rather than `=` is what makes both shapes work; a bare symbol still behaves / exactly as before, so every existing caller is unaffected + requireinit[`getservers]; if[not 11h=abs type pt;raiseerror[`getservers;"proctype must be a symbol or symbol list"]]; $[`~pt; select from .z.m.SERVERS where not null w; @@ -221,6 +319,7 @@ updatestats:{[wh] gethandlebytype:{[pt;selection] / get a single live handle for a proctype via a selection algorithm (`any`roundrobin`last), or / 0Ni if none is connected. bumps usage stats on the chosen row. + requireinit[`gethandlebytype]; if[not -11h=type pt;raiseerror[`gethandlebytype;"proctype must be a symbol"]]; if[not -11h=type selection;raiseerror[`gethandlebytype;"selection must be a symbol (`any`roundrobin`last)"]]; r:getservers[pt]; @@ -243,6 +342,8 @@ waitfortype:{[pt;timeoutms;pollms] / sleeping pollms. returns 1b once connected, 0b on timeout - the CALLER decides if that is fatal. / NOTE the blocking system"sleep" is fine at startup (single-threaded; the injected timer's .z.ts / just doesn't fire during the sleep). + / a zero or already-expired timeout performs NO active retry - see servers.md, waitfortype[pt;0;p] + requireinit[`waitfortype]; if[not -11h=type pt;raiseerror[`waitfortype;"proctype must be a symbol"]]; if[not (abs type timeoutms) within 5 7h;raiseerror[`waitfortype;"timeoutms must be an integer (ms)"]]; if[not (abs type pollms) within 5 7h;raiseerror[`waitfortype;"pollms must be an integer (ms)"]]; @@ -262,6 +363,7 @@ getapimeta:{[] / di.api. init/getapimeta are plumbing (di.torq calls them by convention) and are deliberately NOT / listed - the registry describes the callable api, not plumbing. names are bare (di.torq qualifies). :flip `name`public`descrip`params`return!flip( + (`teardown; 1b; "release the .z.pc handler and retry job installed by init"; "[]"; "null"); (`startup; 1b; "open connections to configured proctypes from process.csv (reads init config)"; "[]"; "null"); (`getservers; 1b; "live SERVERS rows for a proctype, a list of proctypes, or ` for all"; "[symbol|list: proctype, or ` for all]"; "table: live server rows"); (`gethandlebytype; 1b; "one live handle for a proctype via any/roundrobin/last selection"; "[symbol: proctype; symbol: selection]"; "int: handle, 0Ni if none"); diff --git a/di/servers/test.csv b/di/servers/test.csv index 11f13c66..a4fd3146 100644 --- a/di/servers/test.csv +++ b/di/servers/test.csv @@ -1,15 +1,74 @@ action,ms,bytes,lang,code,repeat,minver,comment before,0,0,q,svc:use`di.servers,1,1,load the module -before,0,0,q,os:use`di.os,1,1,di.os for abspath (test harness only) -before,0,0,q,"system ""l "", os.abspath[""di/servers/test.q""]",1,1,load recording mocks + real-peer fixture helpers +comment,,,,,,,"paths are resolved from the MODULE directory via .Q.m.mp, not from the current working" +comment,,,,,,,"directory. di.os.abspath[""di/servers/test.q""] resolved against CWD, so the suite only ran from" +comment,,,,,,,"the repo root - from anywhere else it died with di//di/servers/test.q: No such file, which" +comment,,,,,,,"reads like a missing fixture rather than a path bug. .Q.m.mp is what k4unit itself uses to find" +comment,,,,,,,this very test.csv +before,0,0,q,".t.MODDIR:.Q.m.mp`di.servers",1,1,this module's own directory - independent of where q was started +before,0,0,q,"system ""l "", .t.MODDIR,""/test.q""",1,1,load recording mocks + real-peer fixture helpers before,0,0,q,setupfixture[],1,1,write a temp process.csv fixture (self/otherproc/deadproc) and pick free ports before,0,0,q,spawnpeer[],1,1,launch a genuinely separate q peer to act as otherproc +comment,,,,,,,"requireinit - every exported function except getapimeta refuses to run before init. these rows" +comment,,,,,,,"MUST stay above EVERY init attempt below, including the failing ones: an init validation gap" +comment,,,,,,,"would let one of those rows succeed and wire the logger, and these rows would then fail for that" +comment,,,,,,,"reason instead of a missing guard (measured). they assert the NAMED message rather than just" +comment,,,,,,,"that something threw: a bare fail row would pass on any unrelated throw, and a row checking only" +comment,,,,,,,"""returned 0/empty"" would still pass under the old silent behaviour and prove nothing" +run,0,0,q,"estartup:@[{svc[`startup][]};::;{[e] e}]",1,1,call startup before init and capture the error text +true,0,0,q,"0 / cleanup, tested in isolation from the auto .z.pc hook (which di.handlers would install for real). handlercalls:([]event:`symbol$();name:`symbol$()); +/ remove DELETES the recorded row (it used to be a no-op), so teardown's release is observable the +/ same way register's is - a no-op remove would let a broken teardown pass every assertion. mockhandlers:`register`remove`list!( {[ev;ph;nm;pri;fn]`handlercalls upsert(ev;nm)}; - {[ev;ph;nm]}; + {[ev;ph;nm] handlercalls::delete from handlercalls where event=ev,name=nm;}; {[ev]}); warnlogged:{[s] any (exec msg from logrows where lvl=`warn) like "*",s,"*"};