This project contains the compiler engine written in Python. The compiler engine is responsible for compiling and processing submissions by pooling over the database and updating the results directly on the database.
The recommended way to run and build the project is by using Docker Compose. To use this method, you need to have Docker and Docker Compose installed. If you have any doubts on how to do it, follow the official guide for Docker and the official guide for Docker Compose. Please note that you need to mount /var/run/docker.sock to use it through Docker.
The project's configuration is done through environment variables, which can be checked on the rcc/config.py file.
The engine processes commits through two nested knobs:
RUNCODES_COMPILER_NUM_WORKERS(default2): the number of consumer tasks running on the engine's single event loop. Each consumer owns a share of the concurrency below.RUNCODES_COMPILER_CONCURRENCY(default4): the number of commits each consumer processes concurrently.
The total number of in-flight commits is the product of the two
(num_workers × concurrency, default 2×4 = 8). The same values are read from
JSON configuration files through the num_workers and concurrency_per_worker
keys (see config/rcc/config.json.example).
The workload is IO-bound (containers, S3, database), so sizing has nothing to do with the CPU count: the real ceiling is how many compilation containers the Docker host can run at once, plus available RAM. The whole engine runs in a single process (threads would not add parallelism for IO-bound work), so when the host can take more in-flight work, raise the concurrency and/or the number of consumer tasks.
On startup the engine validates the values (num_workers >= 1,
concurrency >= 1, and a bounded task queue at least as large as the total
number of in-flight slots), refuses to start on nonsensical values and logs
one line with the effective parallelism (e.g. workers=2, concurrency=4, max_in_flight=8).
The engine owns a single psycopg_pool connection pool, tuned through:
RUNCODES_DB_POOL_MIN_SIZE(default1)RUNCODES_DB_POOL_MAX_SIZE— when not set, derived from the total in-flight slots asnum_workers × concurrency + 2(clamped to at least the minimum size); an explicitly configured value always winsRUNCODES_DB_POOL_TIMEOUT(default30seconds)
One pooled connection per in-flight commit is enough because a commit only
holds a connection for short DB bursts (a single transaction per provider
call); the +2 margin covers transient overlap between a finishing commit
and the next one starting.
The Compiler-Engine does not provide an API for external access. The entry point of the application is the
rcc package (on the __init__.py). The recommended execution method is through Docker Compose, even though
uv is used to manage dependencies.
For information on the license of this project, please see our license file.
For information of the contributors of this project, please see our contributors file.
For information on contributing to this project, please see our contribution guidelines.