Concept Guidance

Precise, Training-Free Latent Control for Text-to-Image Generation

Nikolai Röhrich^1,2,* · Isabell Hans^1,2,* · Felix Krause^3,4,* · Björn Ommer^3,4

¹ LMU Munich · ² Konrad Zuse School of Excellence in Reliable AI (relAI) · ³ CompVis @ LMU Munich · ⁴ Munich Center for Machine Learning (MCML) · ^* Equal contribution

Overview

Text-to-image diffusion models offer little direct control over continuous, concept-specific properties and can be unreliable when a prompt requires strong local coherence, such as legible text or anatomically plausible hands.

Concept Guidance (CoG) is a precise, target-specific inference method based on the observation that individual transformer layers contribute very differently to distinct visual concepts. The paper studies this localization through concept-wise mutual information; the released CoG configurations are obtained by profiling each layer on the target metric. During generation, CoG reinforces the relevant layers by combining predictions in which each one is skipped:

$$\epsilon_{\mathrm{neg}} = \frac{\sum_{\ell \in \mathcal{L}_c} w_\ell\,\epsilon_{-\ell}} {\sum_{\ell \in \mathcal{L}_c} w_\ell}$$ $$\epsilon_{\mathrm{CoG}} = (1-\lambda)\epsilon_{\mathrm{neg}}+\lambda\epsilon_\theta = \epsilon_\theta+(\lambda-1) \left(\epsilon_\theta-\epsilon_{\mathrm{neg}}\right).$$

Here, ℒ_c is the set of layers relevant to concept c, w_ℓ is a layer's measured target importance, ε_θ is the model's standard prediction (including CFG where applicable), and λ is the continuous Concept Guidance scale. At λ = 1, CoG reduces exactly to the standard prediction.

CoG works with pretrained models out of the box and requires:

• no training or fine-tuning,
• no external model or reward function during generation,
• no gradients during inference, and
• no prompt engineering.

Offline profiling of a new target does require a target metric or VLM judge once; inference with an existing preset does not. The repository supports PixArt-α, Stable Diffusion 3, Stable Diffusion 3.5 Large, and FLUX.1-dev.

Quick start

Clone the repository and install the runtime dependencies:

cd Concept_Guidance

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

The custom pipelines track the Diffusers 0.35.1 API. Model weights are downloaded from Hugging Face on first use.

The following complete example applies the paper's PixArt-α aesthetics preset:

import torch

from src.PixartAlpha_custom_pipeline import PixArtAlphaPipeline
from src.concept_configs import MODEL_IDS, get_preset

preset = get_preset("pixart", "aesthetics")

pipe = PixArtAlphaPipeline.from_pretrained(
    MODEL_IDS["pixart"],
    torch_dtype=torch.float16,
)
pipe.enable_model_cpu_offload()

# Weighted layer-wise predictions used by Concept Guidance.
pipe.multiskip = True
pipe.skipped_layers = list(preset.layers)
pipe.layer_weights = list(preset.weights)

image = pipe(
    prompt=(
        "a model holding a perfume bottle"
    ),
    guidance_scale=3.5,
    skip_layer_guidance_scale=2.5,
    generator=torch.Generator("cpu").manual_seed(999),
).images[0]

image.save("concept_guidance.png")

For a same-seed baseline, reset pipe.skipped_layers = None and omit skip_layer_guidance_scale.

Tutorial notebook

The Concept Guidance tutorial is the recommended entry point. It includes:

• one-line model selection across all four supported families,
• all configurations reported in Tables 6 and 11, plus retained research presets,
• same-prompt, same-seed baseline comparisons,
• a continuous guidance-scale sweep,

Launch it from the repository root:

pip install jupyterlab
jupyter lab examples/concept_guidance.ipynb

Supported models and concepts

Key	Checkpoint	Custom pipeline	Paper targets	Additional retained targets
pixart	PixArt-alpha/PixArt-XL-2-1024-MS	PixArtAlphaPipeline	hands, aesthetics	text, counting, position, hands + aesthetics
sd3	stabilityai/stable-diffusion-3-medium-diffusers	StableDiffusion3Pipeline	visible text, hands, aesthetics	counting, position, combined concepts
sd35	stabilityai/stable-diffusion-3.5-large	StableDiffusion3Pipeline	visible text, hands, aesthetics	counting, position, symmetry, ukiyo-e, background separation, combined concepts
flux	black-forest-labs/FLUX.1-dev	FluxPipeline	visible text, hands, aesthetics	counting, position, combined concepts

SD3, SD3.5, and FLUX.1-dev are gated. Accept the corresponding Hugging Face license and authenticate before loading them:

hf auth login

Each checkpoint remains subject to its own license and acceptable-use policy.

Paper configurations

Presets are saved in src/concept_configs.py. The following values reproduce Table 6 (layers and normalized weights) and the selected settings in Table 11 (guidance scale λ). They are exposed under the canonical keys text, hands, and aesthetics in PAPER_PRESETS and CONCEPT_PRESETS.

Model	Target	Layers	Weights	λ
PixArt-α	Human hands	[3, 23, 8, 18]	[1.00, 0.88, 0.73, 0.63]	2.00
PixArt-α	Aesthetics	[9, 20, 5]	[1.00, 0.28, 0.12]	2.50
SD3	Visible text	[7, 11, 4]	[1.00, 0.55, 0.45]	2.25
SD3	Human hands	[9, 6, 5, 10]	[1.00, 0.48, 0.45, 0.21]	2.00
SD3	Aesthetics	[6, 7, 8]	[1.00, 0.71, 0.70]	2.50
SD3.5 Large	Visible text	[15, 10]	[1.00, 0.49]	2.50
SD3.5 Large	Human hands	[7, 18]	[1.00, 0.98]	2.25
SD3.5 Large	Aesthetics	[4]	[1.00]	2.00
FLUX.1-dev	Visible text	[11, 3, 9]	[1.00, 0.85, 0.74]	2.25
FLUX.1-dev	Human hands	[9, 12, 15]	[1.00, 0.81, 0.78]	1.75
FLUX.1-dev	Aesthetics	[15, 14, 13]	[1.00, 0.18, 0.09]	3.00

Other entries in CONCEPT_PRESETS are retained from the research repository for convenience. They are intentionally excluded from PAPER_PRESETS because they are not configurations listed in Table 6.

The paper reports that PixArt-α does not generate visible text; its retained text entry is therefore an additional research preset, not a paper benchmark configuration.

Model-specific usage

All presets are returned through the same interface:

from src.concept_configs import get_preset

preset = get_preset("sd35", "text")
print(preset.layers)
print(preset.weights)
print(preset.scale)

The final pipeline arguments differ slightly because the custom classes preserve their upstream Diffusers APIs.

Stable Diffusion 3 / 3.5

import torch

from src.SD3_custom_pipeline import StableDiffusion3Pipeline
from src.concept_configs import MODEL_IDS, get_preset

model = "sd35"
preset = get_preset(model, "text")

pipe = StableDiffusion3Pipeline.from_pretrained(
    MODEL_IDS[model],
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()
pipe.multiskip = True
pipe.cfg_skip = False  # Algorithm 2 uses conditional skipped predictions.
pipe.layer_weights = list(preset.weights)

image = pipe(
    prompt="A bakery storefront with a sign that reads 'MORNING BREAD'",
    skip_guidance_layers=list(preset.layers),
    skip_layer_guidance_scale=preset.scale,
    skip_layer_guidance_start=0.0,
    skip_layer_guidance_stop=1.0,
    generator=torch.Generator("cpu").manual_seed(42),
).images[0]

FLUX.1-dev

import torch

from src.FLUX_custom_pipeline import FluxPipeline
from src.concept_configs import MODEL_IDS, get_preset

preset = get_preset("flux", "hands")

pipe = FluxPipeline.from_pretrained(
    MODEL_IDS["flux"],
    torch_dtype=torch.bfloat16,
)
pipe.enable_model_cpu_offload()
pipe.multiskip = True
pipe.skipped_layers = list(preset.layers)
pipe.layer_weights = list(preset.weights)

prompt = "Close-up photograph of a ceramic artist shaping a bowl with both hands"
image = pipe(
    prompt=prompt,
    negative_prompt=prompt,
    guidance_scale=3.5,
    true_cfg_scale=preset.scale,
    generator=torch.Generator("cpu").manual_seed(42),
).images[0]

Using the same text in the regular and skipped branches is intentional: the contrast comes from skipping concept-relevant layers, not from changing the prompt.

PixArt-α

PixArt-α follows the quick-start example. Set:

pipe.multiskip = True
pipe.skipped_layers = list(preset.layers)
pipe.layer_weights = list(preset.weights)

and pass skip_layer_guidance_scale=preset.scale to the pipeline call.

Tuning the control strength

The paper preset scale is the recommended starting point. For a new prompt, keep the seed fixed and sweep around it:

scales = [1.0, 1.5, 1.75, 2.0, 2.25, 2.5, 3.0]

• 1.0 is the baseline endpoint.
• Larger values strengthen the selected concept.
• Excessive guidance can overshoot and introduce artifacts.

The paper evaluates λ from 1.25 to 3.00 in increments of 0.25; Table 11 selects the task-specific values encoded above.

Concept Guidance adds one transformer evaluation per selected layer at each guided denoising step. Using fewer top-ranked layers reduces inference time.

Repository structure

.
├── examples/
│   └── concept_guidance.ipynb    # End-to-end tutorial
├── tests/
│   └── test_paper_presets.py     # Guards the published configurations
├── src/
│   ├── concept_configs.py        # Paper configurations + extra research presets
│   ├── FLUX_custom_pipeline.py
│   ├── FLUX_custom_transformer.py
│   ├── PixartAlpha_custom_pipeline.py
│   └── SD3_custom_pipeline.py
├── prompt_datasets/              # Concept-specific prompt pairs
├── eval/                         # Evaluation helpers
├── CITATION.cff                  # GitHub-native citation metadata
├── requirements.txt              # Minimal inference dependencies
└── requirements-eval.txt         # Optional evaluation dependencies

To use the evaluation helpers, install their optional dependencies:

pip install -r requirements-eval.txt

Adding a new concept

Algorithm 1 in the paper profiles a new target once per model:

1. Collect a representative prompt set and choose a target metric or VLM judge.
2. Measure baseline target performance over the prompt set.
3. Repeat generation while skipping one layer at a time at profiling scale 2.0.
4. Set each weight to the positive performance gain over baseline, select the top-k layers, and tune only λ.

The paper's concept-wise mutual-information analysis explains why these layer effects are target-specific, but MI computation is not required to use CoG. After profiling, ordinary generation remains training-free and requires only the custom pipeline. Add the resulting values to src/concept_configs.py, or assign pipe.layer_weights and the relevant layer indices directly.

Citation

If you find this work useful, please cite:

@inproceedings{roehrich2026concept,
  title     = {Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation},
  author    = {R{\"o}hrich, Nikolai and Hans, Isabell and Krause, Felix and Ommer, Bj{\"o}rn},
  booktitle = {German Conference on Pattern Recognition (GCPR)},
  year      = {2026}
}

The final proceedings and paper links will be added when they become available.

Acknowledgements

The custom pipelines build on Hugging Face Diffusers. We thank the authors of PixArt-α, Stable Diffusion 3/3.5, and FLUX.1 for releasing their models and inference code.