Tangor

PoC of StarPU based Iyokan

CPU build

Tangor builds the CPU path by default. The HOGE/FPGA path remains opt-in via -DUSE_HOGE=ON.

git submodule update --init --recursive
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j

Tangor uses thirdparties/cuFHEpp for the TFHEpp checkout at thirdparties/cuFHEpp/thirdparties/TFHEpp. It does not build cuFHEpp's CUDA library for the CPU path.

The default TFHE configuration is Block Binary keys with Subset Keys (USE_BLOCK_BINARY=ON, USE_SUBSET_KEY=ON). Regenerate secret/evaluation keys, encrypted packets, and snapshots after upgrading, since artifacts made with the former defaults are not compatible.

KVSP compatibility

With the default -DTANGOR_BUILD_KVSP_COMPAT=ON, the build produces build/bin/iyokan and build/bin/iyokan-packet in addition to Tangor's native executable. They implement the CLI and cereal packet/archive format expected by KVSP, including plain, tfhe, snapshots, key generation, and packet conversion. Point KVSP at them without changing the KVSP command line:

KVSP_IYOKAN_PATH="$PWD/build/bin/iyokan" \
KVSP_IYOKAN_PACKET_PATH="$PWD/build/bin/iyokan-packet" \
  /path/to/kvsp

The compatibility targets share the checked-out Iyokan frontend sources while linking to Tangor's selected TFHEpp build. This keeps packets, evaluation keys, and snapshots byte-compatible during the scheduler migration. Set -DTANGOR_BUILD_KVSP_COMPAT=OFF for a standalone Tangor-only build. If that checkout is not adjacent to Tangor, CMake fetches the pinned Iyokan frontend (including its submodules) by default; use -DTANGOR_FETCH_IYOKAN_COMPAT=OFF plus TANGOR_IYOKAN_COMPAT_SOURCE_DIR and TANGOR_IYOKAN_COMPAT_THIRDPARTY_DIR for an offline source mirror.

StarPU CPU/GPU execution

Tangor schedules the KVSP gate dependency graph with StarPU. In CUDA mode, StarPU owns each gate's read/write dependency while an event-driven cuFHEpp stream pool evaluates the bootstrap and releases the dependent task only after the result is back on the host. This provides SM-scale CUDA concurrency without one host thread per CUDA stream. ROM/RAM cell refresh uses the same CUDA-pool model. The ROM/RAM FFT CMUX tree remains a TFHEpp CPU operation, matching Iyokan's proven path; TANGOR_EXPERIMENTAL_CUDA_CMUX=1 is retained only for kernel-integration development. No TFHE parameters are changed for runtime compatibility. CPU and CUDA builds both use the split Fourier archive layout, so keys and encrypted ROM/RAM packets are portable across Tangor, Iyokan, and KVSP.

Tangor accepts KVSP's Iyokan CMake cache variables, so KVSP can use Tangor as its source directory without renaming build flags:

git submodule update --init --recursive
cmake -S /path/to/Tangor -B build/Iyokan-avx2 \
  -DCMAKE_BUILD_TYPE=Release \
  -DIYOKAN_ENABLE_CUDA=ON \
  -DIYOKAN_MARCH=x86-64-v3 \
  -DUSE_AVX512=OFF
cmake --build build/Iyokan-avx2 --target iyokan iyokan-packet

To have KVSP build that compatibility target in its normal location:

make -C /path/to/kvsp ENABLE_CUDA=1 IYOKAN_SOURCE=/path/to/Tangor iyokan-avx2

Run with both CPU and GPU resources. For example, this uses 64 CPU workers and two GPUs:

/path/to/kvsp/build/bin/iyokan-avx2 tfhe --enable-gpu --cpu 64 --num-gpu 2 \
  --evalkey eval.key -c 224 -o result.enc --snapshot result.snapshot \
  --blueprint /path/to/kvsp/build/share/kvsp/alexandrite.toml -i fib.enc

--cpu selects the physical StarPU CPU-worker budget, --num-gpu selects CUDA devices, and --enable-gpu enables the CUDA path. Tangor defaults to the physical CPU-core count for CPU-only execution and to all logical CPUs in CUDA mode, where those threads feed independent RAM/ROM preparation and launches. CUDA mode uses StarPU's eager scheduler by default; CPU-only mode uses dmdas. STARPU_SCHED remains an override. The CUDA pools default to two gate streams per SM (TANGOR_GATE_CUDA_WORKERS_PER_GPU) and one RAM-refresh stream per SM (TANGOR_RAM_CUDA_WORKERS_PER_GPU). STARPU_NWORKER_PER_CUDA is left at one worker per device by default because it is a host thread, not a CUDA stream. Each CUDA device has its own event dispatcher, avoiding device-context switches in the hot polling and copy paths. CPU-only execution uses the same graph without CUDA workers.

CUDA MUX gates can use contiguous, asynchronous batches by setting TANGOR_MUX_CUDA_BATCH_SIZE to a positive integer. The default is 1, which selects the legacy single-MUX launch path; on the tested two-A100 system, larger batches improved Chrysoberyl by less than the required 5% default-change threshold. A partial batch launches immediately—the dispatcher does not delay ready work merely to fill it. Binary and MUX launches share the same two-blocks-per-SM device budget.

RAM-write CMUX chunking is available for performance experiments through IYOKAN_RAM_CMUX_CHUNK_SIZE. A positive value groups that many contiguous RAM words into one CPU task and submits their GPU refreshes as one aggregate task; 0 selects the original one-task-per-word graph and is the default. The RAM read reduction tree is unchanged.

StarPU diagnostics

Profiling is off by default. Build with -DTANGOR_STARPU_PROFILE=ON to enable worker/per-task reports and FxT tracing by default. This requires the fxt development package; environment variables still override the defaults, so reports can be redirected or disabled without rebuilding.

Set TANGOR_RUNTIME_PROFILE=1 on any build to report the work performed outside native StarPU codelets: CPU CMUX chains, circuit bootstrapping, sample-extract/key-switch work, and queue/service latency for the custom CUDA gate and RAM-refresh pools. It can be combined with STARPU_PROFILING=1 and STARPU_WORKER_STATS=1 for a complete host-side report. Profiling adds timing and atomic-counter overhead and is therefore disabled by default.

Use --critical-profile-prefix PATH when the dependency path and resource pressure must be correlated for each emulated cycle. The evaluator writes PATH-<cycle>.json and prints compact CPROF: lines. The JSON contains the causal Iyokan graph (late DFF feedback is marked noncausal), the observed releasing-dependency chain, CPU time, StarPU operations, CUDA gate/RAM queue latency, and per-device CUPTI kernel and copy activity. Nodes are attributed to core:<name>, ram:<name>, rom:<name>, or bridge domains so total work can be compared with critical-path work directly. Profiles with dropped or uncorrelated runtime records are marked incomplete and do not make a tuning recommendation.

CUPTI support is built by default in CUDA configurations and can be disabled with -DTANGOR_ENABLE_CUPTI_PROFILE=OFF. A GPU run that requests critical profiling fails early if its binary lacks CUPTI support; CPU-only critical profiling remains available.

The evaluator cannot inspect encrypted termination itself. For KVSP's bundled fib(5) input, a plaintext emulator establishes that 224 cycles are required. Decrypt the resulting packet to confirm f0 = true and x10 = 5:

/path/to/kvsp/build/bin/kvsp dec --cpu alexandrite -k secret.key -i result.enc

StarPU gate scheduling is enabled by default. It can be stated explicitly for reproducible builds:

cmake -S /path/to/Tangor -B build/starpu \
  -DIYOKAN_ENABLE_CUDA=ON -DTANGOR_KVSP_STARPU_GATE_OFFLOAD=ON
build/starpu/bin/iyokan tfhe --enable-gpu --cpu 64 --num-gpu 2 ...

To use a different cuFHEpp checkout:

cmake -S . -B build -DTANGOR_CUFHEPP_SOURCE_DIR=/path/to/cuFHEpp

To override only TFHEpp:

cmake -S . -B build -DTANGOR_TFHEPP_SOURCE_DIR=/path/to/TFHEpp