feat(signage-ai): provider and job models (PPT-2741) - #326
Merged
Conversation
Two tables behind AI generated signage artwork: - signage_ai_providers holds vendor credentials per domain, encrypted at rest and never rendered by the API. A row with no authority is the shared fallback, the arrangement storages already uses, and a trigger keeps one default per domain. - signage_ai_jobs is one generate or edit request. Candidates are written into result->images one at a time by an atomic jsonb_set so concurrent writers cannot drop each other, and every write bumps a version counter for the long polling endpoint to compare against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ating usage Both quota sums excluded failed jobs, so a caller whose requests kept failing had no limit at all, which is the one case where a limit matters most. Most failures reached the vendor and were billed. `attach_item` records that a candidate became a media item without the images_produced increment welded into `bump_image`. Claiming is not producing, and counting it twice inflated the usage report by every save.
Copied from bump_image without the CASE that creates the array when it is absent. Unreachable from today's callers, which only get there after matching an existing entry, but the version would still bump while the write silently did nothing.
Steve asked for automatic cleanup so a busy domain does not accumulate its
whole history. A before_create callback on the job, so it runs as each new
job arrives rather than needing a scheduler.
Off unless SIGNAGE_AI_CLEANUP_DAYS is set to a positive number of days.
Nothing starts deleting a site's history because it was upgraded, and
somewhere that generates seasonally keeps last year's work by doing nothing.
Zero and negatives both read as "keep everything" rather than as a cutoff in
the future that would take the whole table with it.
The delete runs in a nested transaction, which PgORM issues as a SAVEPOINT.
before_create runs inside the transaction that inserts the new row, so
without that a failed cleanup would abort the insert and a user would fail to
generate an image because tidying up went wrong. Rescuing alone is not
enough: Postgres has already marked the transaction aborted by then.
It reads the ids and then deletes them, rather than the one statement this
obviously wants to be:
DELETE FROM signage_ai_jobs WHERE id IN (
SELECT id FROM signage_ai_jobs WHERE created_at < $1
ORDER BY created_at LIMIT $2 FOR UPDATE SKIP LOCKED)
That does not reliably delete at most n rows. When the planner picks a nested
loop semi join it re-runs the subquery once per candidate row, and since SKIP
LOCKED and the rows already deleted by the statement change what comes back
each time, the delete lands on the union of those evaluations. Confirmed with
an audit trigger and auto_explain: a hash semi join runs the Limit once and
removes 2 of 3, a nested loop runs it three times and removes all 3. Which
plan you get moves with the table statistics.
Work per insert is capped so a long disabled cleanup being switched on cannot
stall a request while it catches up.
One caveat for operators turning it on: the usage report and the per domain
monthly quota both read this table, so a window shorter than a month makes
both under-count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of the AI signage feature (PPT-2741). This is the bottom of the stack: everything else waits on it.
What this adds
Two models and the migration behind them.
SignageAIProviderholds a vendor configuration: which vendor, which model, the endpoint and region, whether it is enabled, which one is the default, and the per-user and per-domain quotas. Credentials are encrypted at rest and never come back out over the API. A provider can be scoped to a single domain or left global.SignageAIJobis one generation or edit request and the images it produced. It carries the prompt, the state, the candidates, and enough accounting to answer "has this user run out" without a second table.Notes for review
shard.ymlis deliberately untouched so the version bump automation can pick the increment from the commit types. These arefeatandfix, so it should land on 9.109.0.Quota accounting counts failed jobs. That is intentional: a failed job still costs a vendor call, and not counting them turns a retry loop into a free pass. Claiming an image into the media library does not add to usage, since the job that produced it already did.
One thing worth a second opinion: the two enum columns are stored as
TEXTwith aCHECKconstraint rather than a native PostgresENUM. Every otherPGEnumConvertercolumn in this repo (17 of them) usesCREATE TYPE.TEXTplusCHECKis easier to extend later, but it is a third mechanism in a repo that had two, and I would rather that be a decision than an accident. Happy to convert it.Retention (added after review feedback)
Steve asked for automatic cleanup of old job rows, so there is now a
before_createcallback that prunes past a retention window as each new job arrives.It ships off. Nothing happens unless
SIGNAGE_AI_CLEANUP_DAYSis set to a positive number of days, so an upgrade never starts deleting a site's history, and somewhere that generates seasonally keeps last year's work by doing nothing. Zero and negatives both read as "keep everything".Two things in there worth a reviewer's attention.
The delete runs in a nested transaction, which PgORM issues as a
SAVEPOINT.before_createruns inside the transaction that inserts the new row, so without it a failed cleanup would abort that insert and a user would fail to generate an image because housekeeping went wrong. A plainrescuedoes not help, since Postgres has already marked the transaction aborted.It reads the ids and then deletes them, rather than the single statement this obviously wants to be:
That does not reliably delete at most n rows. When the planner picks a nested loop semi join it re-runs the subquery once per candidate row, and because
SKIP LOCKEDand the rows already deleted by the statement change what comes back each time, the delete lands on the union of those evaluations. Confirmed with an audit trigger andauto_explain: a hash semi join runs the Limit once and removes 2 of 3, a nested loop runs it three times and removes all 3. The plan choice moves with table statistics, so it passes in psql and in tests until it does not.Tested end to end against a local stack with the variable unset, at 90 days and at 30 days, plus the model specs and the 223 rest-api signage controller specs.
One caveat for operators turning it on: the usage report and the per domain monthly quota both read this table, so a window shorter than a month makes both under-count.
Merge order
This one first, and it needs to be tagged before rest-api can pin it. Then rest-api, then backoffice. ts-client, user-interfaces and placeos-docs have no ordering constraint.
Testing
Specs for both models, including the quota arithmetic and the guard against a job with no images array.