fix(demo): produce a real report instead of hanging, then an empty one

[kmadan]

Closes #74.

aicertify demo did not finish. I killed a run at 1156s still sitting on Starting evaluation against eu_ai_act. It now completes in about two minutes and writes a report with actual verdicts.

	Before	After
Completes	never (killed at 19m 16s)	119s, exit 0
Report	Total Policies: 0	Total Policies: 4, Red: 4
Billable calls with a key set	yes, silently	no, unless --with-llm-metrics

Three defects, each hiding the next

1. It blocked indefinitely on an LLM call. py-spy on the live process showed both threads idle in select(), the worker inside deepeval/metrics/toxicity/toxicity.py. DeepEval's ToxicityMetric is LLM-judged, so measure() is a network call — with no bound.

ContentSafetyEvaluator now runs each interaction under a wall-clock bound (llm_timeout_seconds, default 60). A timeout fails closed — worst-case toxicity, passed=False — matching the existing unavailable path. A content-safety check that did not complete has established nothing, and reporting it clean would be the same fail-open this library has closed three times in the policies.

The worker is a daemon thread, not a ThreadPoolExecutor, and that distinction is load-bearing: executor threads are non-daemon and the interpreter joins them at exit, so a call still blocked in the provider would keep the process alive after we stopped waiting for it, defeating the bound. Measured — abandoning a 300s call adds no exit delay.

2. It made billable calls uninvited. The LLM-judged evaluators activate off the presence of OPENAI_API_KEY. So the demo advertised as needing "no API keys" silently became a paid, minutes-long run for anyone with a key exported — which is most developers.

It now hides that variable for its own run unless --with-llm-metrics is passed, restores it in a finally, and says which mode it is in. This uses the skip-gracefully path the runner's own docstring already described.

3. Then it produced an empty report and called it success. With the hang gone, the report read Total Policies: 0 while exiting 0 — arguably worse than hanging, because it looks like it worked.

The cause was mine. opa eval --data <dir> parses every .yaml and .json in the tree as a data document, and the GitHub issue-template forms I added to gopal in an earlier session collide:

.github/ISSUE_TEMPLATE/new_framework.yml: merge error

OPA exited 2 having evaluated nothing, on all eleven policy groups, and the empty result set was rendered as a clean report rather than an error. gopal's own CI passes these ignores to opa check and opa test; all three opa eval call sites here had none. They now share opa_ignore_flags() so a fourth site cannot silently omit them, and a test asserts exactly that.

About that 4

The report shows 4 policies, not 29. That is a pre-existing interface gap, not a remnant of this fix. Only 4 of 98 gopal policies define the report_output rule AICertify queries — and it was 4 of 87 before the submodule bump, so the bump did not cause it:

54dfbea: 4 of 87 policies define report_output
main:    4 of 98 policies define report_output

Most policies expose compliance_report instead. Worth its own issue: closing it would take the demo from 4 verdicts to 29, and is probably the single largest remaining gap between what gopal encodes and what AICertify can report.

Also

The demo now ends by naming the two commands that answer "what would make these pass?":

Expect denials: the bundled contract declares no compliance evidence, and most
obligations turn on facts no evaluator can observe. To see what eu_ai_act
actually asks for, and to scaffold a contract that answers it:
    aicertify explain eu_ai_act
    aicertify init-contract --policy eu_ai_act > contract.json

A column of unexplained denials reads like a broken tool. This connects the demo to the discovery commands from #73.

The README's timing note is corrected to ~2 minutes with an explanation, replacing the "~10 seconds" it claimed before #73.

Tests

25 new, 88 total. The ignore guard is mutation-verified: removing the flags from one of the three call sites turns it red. The daemon-thread property is asserted directly, since a ThreadPoolExecutor would pass every other test in that file while silently reintroducing the hang at exit.

[Prinevo]

Approved