Structured Generative Dreaming for Continual Learning

Salvatore Calcagno, Matteo Pennisi, Federica Proietto Salanitri, Amelia Sorrenti, Simone Palazzo, Concetto Spampinato and Giovanni Bellitto

Overview

This is the official PyTorch implementation for paper "Dream2Learn: Structured Generative Dreaming for Continual Learning".

Abstract

Continual learning requires balancing plasticity and stability while mitigating catastrophic forgetting.

Inspired by the concept of human dreaming as a source of internal simulation and knowledge restructuring, we introduce Dream2Learn (D2L), a framework in which a continual classifier leverages its internal representations to guide the generation of structured synthetic experiences and uses them for self-improvement.

Rather than reconstructing past data as in generative replay, D2L enables a classifier to create novel, semantically distinct dreamed classes that are coherent with its learned knowledge yet do not correspond to previously observed data. These dreamed samples are produced by conditioning a frozen diffusion model through soft prompt optimization driven by the classifier itself. The generated data are not used to replace memory, but to expand and reorganize the representation space, effectively allowing the network to leverage internally synthesized auxiliary concepts during training. By integrating dreamed classes into continual training, D2L proactively structures latent features to support forward knowledge transfer and adaptation to future tasks. This prospective self-training mechanism conceptually relates to the role of sleep in consolidating and reorganizing memory, turning internal simulations into a tool for improved generalization. Experiments on Mini-ImageNet, FG-ImageNet, and ImageNet-R demonstrate that D2L consistently outperforms strong rehearsal-based baselines and improves forward transfer, achieving positive values in several of the evaluated settings, confirming its ability to enhance adaptability through internally generated training signals.

Method

Installation

Python 3.9 and a CUDA 12.1-compatible system are expected.

uv venv --python 3.9
source .venv/bin/activate
uv pip install -r requirements.txt

Datasets

The datasets are not redistributed. See DATASETS.md for official sources, exact released splits, expected layouts, and preparation commands.

Run

Run one of the released configurations:

python scripts/dream.py --config configs/miniimg.yaml

• MiniImageNet: configs/miniimg.yaml
• ImageNet-FG: configs/fg_imagenet.yaml
• ImageNet-R: configs/imagenet_r.yaml

Each run executes, task by task:

1. continual training;
2. soft-prompt optimization with the released oracle;
3. dream generation;
4. online dataset update.

Note

Re-running the same configuration resumes matching completed stages. Use a new code value in the YAML after changing an experiment. GPU assignments and W&B behavior can be configured from the command line; see python scripts/dream.py --help.

Verify the released oracle weights:

cd continual_prompt_optimization/oracle_net
sha256sum --check SHA256SUMS

Parts of this repository are adapted from Mammoth.

Citation

@article{calcagno2026dream2learn,
  title   = {Dream2Learn: Structured Generative Dreaming for Continual Learning},
  author  = {Calcagno, Salvatore and Pennisi, Matteo and Proietto Salanitri, Federica and Sorrenti, Amelia and Palazzo, Simone and Spampinato, Concetto and Bellitto, Giovanni},
  journal = {International Journal of Computer Vision},
  volume  = {134},
  number  = {8},
  pages   = {388},
  year    = {2026},
  doi     = {10.1007/s11263-026-02958-1},
  url     = {https://link.springer.com/article/10.1007/s11263-026-02958-1}
}