What is sequifier?

Sequifier is the short, efficient, scalable path from tabular sequences to your own transformer model.

It offers three core commands preprocess, train and infer, each of them configurable, fully parallelised and well-tested.

They enable you to go from multivariate sequence data to a model that ingests such data, and emits either (a) this data, (b) a subset, (c) other variables or (d) embeddings.

If input and target variables are the same, it supports full autoregressive inference.

Through configuration, most modern architectural variants such as RoPE, GQA, SwiGLU, and RMSNorm are supported. The ambition is to follow the frontier when industrial-grade implementations become available.

The process looks like this:

Examples from the literature

Multivariate tabular transformers have many applications.

These independent references illustrate some of them; inclusion does not imply that the cited work used or is affiliated with sequifier.

Finance

• Transaction Foundation Models (Overview, EWE-1)
• Order Book Transformers (TransLOB)
• Volatility Forecasting (paper)
• Macroeconomic Models (BISTRO)

Health/Bio

• Health Records (BEHRT, Med-BERT)
• Treatment Outcome Prediction (G-Transformer)
• Health Trajectory Prediction (ETHOS)
• Glucose Forecasting (GluForecast)
• ECG rhythm/arrhythmia monitoring (Heart Language Model)

Cybersecurity

• Network Intrusion Detection (paper)
• Encrypted Traffic Analysis (Criss-Cross Traffic Transformer)
• Insider Threat Detection (paper)
• In-Vehicle Intrusion Detection (paper)
• IoT Anomaly Detection (paper)
• System Logs Anomaly Detection (DeepEAD)

Industrial & IoT

• Remaining Useful Life Estimation (paper)
• Fault Diagnosis (paper)
• Anomaly Detection for Industrial Control Systems (paper)
• Soft Sensing (Debutanizer)
• Battery State-of-Health Estimation (DS-transformer)
• Production Line Modelling (paper)

Agriculture & Environment

• Soil Moisture Forecasting (paper)
• Crop Water Demand (paper)
• Rainfall–runoff modelling (paper)
• Drought Forecasting (paper)
• Land Surface Dynamics (paper)

Neuroscience

• Neural Population Dynamics (Neural Data Transformer)
• Neural + Motor Modelling (Intracortical Motor Decoder)
• fMRI State Prediction (paper)
• Seizure Detection (BIOT)
• Magnetoencephalography Data (MEG-GPT)

Animal Communication

• Sperm Whales (whale-gpt, WhAM)
• Bengalese Finches (paper)
• Zebra Finches (ZF-Aim)

Sequifier aims to standardise model implementation across these fields, to make them comparable, transfer learnings between domains, and converge on optimal solutions faster.

Value Proposition

For the individual researcher, sequifier cuts the development time of a model significantly: typically, some preprocessing and the subsequent model evaluation are specific to the modelling problem, but all the steps in between are taken care of.

This enables:

• rapid prototyping on a configurable architecture
• trusted implementation (you can't create bugs inadvertently)
• scaling preprocessing across cores and training across GPUs and nodes
• hyperparameter optimization using Optuna (Bayesian, Random, or Grid search)

The Six Commands

There are six standalone commands within sequifier: make, preprocess, train, infer, hyperparameter-search, and visualize-training.

Command	Purpose
make	Create a new sequifier project with config templates.
preprocess	Convert input data into fixed-length subsequences.
train	Train a model on preprocessed data.
infer	Generate predictions, probabilities, or embeddings.
hyperparameter-search	Use Optuna to find optimal configurations across multiple training runs.
visualize-training	Generate interactive HTML plots from structured training metrics.

There are documentation pages for each command, except make:

• preprocess documentation
• train documentation
• infer documentation
• hyperparameter-search documentation
• visualize-training documentation

Other Materials

To get the full documentation, visit sequifier.com

Structure of a Sequifier Project

Sequifier is designed with a specific folder structure in mind:

YOUR_PROJECT_NAME/
├── configs/
│   ├── preprocess.yaml
│   ├── train.yaml
│   └── infer.yaml
├── data/
│   └── (Place your CSV/Parquet files here)
├── models/
├── checkpoints/
├── outputs/
│   ├── embeddings(?)
│   ├── predictions(?)
│   ├── probabilities(?)
│   └── visualization/
├── logs/
├── state/
└── scripts/

The sequifier commands should typically be run in the project root.

Within YOUR_PROJECT_NAME, you can also add other folders for additional steps, such as notebooks or scripts for pre- or postprocessing, and analysis, visualizations or evals for files you generate in other, manual steps.

Data Transformations in Sequifier

The basic input data format is this:

sequenceId	itemPosition	column1	column2	...
0	0	"high"	12.3	...
0	1	"high"	10.2	...
...	...	...	...	...
1	0	"medium"	20.6	...
...	...	...	...	...

The two columns "sequenceId" and "itemPosition" have to be present, and there must be one or more feature columns.

sequifier preprocess splits sequences into subsequences, normalises real variables and maps categorical variables to integers/tokens. The subsequence length is the sum of window_length and max_target_offset.

sequenceId	subsequenceId	startItemPosition	leftPadLength	inputCol	[Subsequence Length - 1]	[Subsequence Length - 2]	...	0
0	0	0	0	column1	"high"	"high"	...	"low"
0	0	0	0	column2	12.3	10.2	...	14.9
...	...	...	...	...	...	...	...	...
1	0	15	0	column1	"medium"	"high"	...	"medium"
1	0	15	0	column2	20.6	18.5	...	21.6
...	...	...	...	...	...	...	...	...

Generative inference returns a row-oriented table with the predicted target columns plus identifiers for the source sequence and model window:

sequenceId	subsequenceId	windowStartOffset	itemPosition	column1	column2	...
0	0	0	963	"medium"	8.9	...
0	0	0	964	"low"	6.3	...
...	...	...	...	...	...	...
1	4	0	732	"medium"	14.4	...
...	...	...	...	...	...	...

Complete Example of Training and Inferring a Transformer Model

Once you have your data in the input format described above, you can train a transformer model in a couple of steps:

1. Install sequifier. Create and activate an environment with Python >=3.10, then run:

   pip install sequifier

2. Create a project. Generate a project folder with config templates in its configs subfolder:

   sequifier make YOUR_PROJECT_NAME

3. Add your data. Change into the YOUR_PROJECT_NAME folder, create a data folder, add your data, and adapt preprocessing_data_path in preprocess.yaml to point to it.

4. Preprocess the data. Run:

   sequifier preprocess

5. Reference the generated metadata. Preprocessing outputs metadata at configs/metadata_configs/[INPUT BASENAME].json. For a single dataset and part, reference that file from dataset.part.metadata_config_path in train.yaml; named configurations use dataset_training.<dataset>.parts.<part>.metadata_config_path. Inference may still use preprocessing_data_path or metadata_config_path.

6. Configure and train the model. Adapt train.yaml with your transformer hyperparameters, then run:

   sequifier train

7. Configure inference. Point model_path in infer.yaml at the default ONNX export. Keep the scaffold's explicit contract, or replace it with training_config_path and dataset; see the ONNX/PT trade-offs.

8. Run inference. Run:

   sequifier infer

9. Find your predictions. They are written to [PROJECT ROOT]/outputs/predictions/[EXPORTED_MODEL_BASENAME]/part-000.[FORMAT], for example outputs/predictions/your-model-best-3/part-000.csv.

Key Features

Causal Modelling

Causal transformers are the core architecture in sequifier. Their training objective is to 'predict the next token', or, in the multivariate case, the next value for each target variable.

To train a causal transformer, set training_objective: causal. If the target value of interest is further in the future, set target_offset to a value higher than 1.

Autoregressive Inference

Autoregressive inference is allowed when the model is causal, all input variables are target variables, target_offset equals 1 and prediction_length equals 1. It is enabled by setting autoregressive: true, and generation_steps to the desired integer value. The number of generation_steps will be generated starting from the first complete subsequence in a sequence.

It iteratively predicts future values, by returning predictions at step t-1 as input for generating a prediction at t. Predictions for categorical target variables can be made using argmax or sampling.

Causal Modelling Variants

Final-value Causal Modelling

In final-value causal models, the final value of each target variable within the subsequence is projected back in time as target. The idea is that the sequence of events leading up to the final value is a continuous accrual of evidence for an outcome, with the final value being the resolution. For example, the sequence of clicks through an online shop are in search of a product, and the product that is actually purchased at the end is the resolution.

Next-occurrence Causal Modelling

Next-occurrence causal modelling is a generalisation of final-value causal modelling: instead of taking the last value of each target variable as target, it takes the next value at a position where another categorical variable matches a criterion value as target. To illustrate this, final-value causal modelling is equivalent to next-occurrence causal modelling where the criterion variable is 'is_last', which is '0' up to the last position, where it is '1', and the criterion value is '1'. The values at the last position are projected 'back' across the subsequence, only now, we also have the option to use a different criterion variable, set it to '1' at multiple locations, and train the model to predict 'next relevant event', rather than just 'last event'.

Causal Embedding Model

sequifier also supports the export of causal embeddings, instead of predictions. It requires the following settings: export_embedding_model: true in the training config and model_type: embedding in the inference config. Selected activations are restricted to the configured final prediction_length positions and concatenated in configuration order along the feature dimension.

If you are interested in activations other than the last backbone layer, you can configure the exact layers you want to contribute to the export using embedding_layer_names. You can pass an ordered list, such as [backbone.layers.1, decoder.branches.default.hidden_blocks.0], and the activations of these layers will be concatenated and output.

Layer names follow the network hierarchy using zero-based indices: backbone.layers.<index> selects a transformer block output, backbone.final_norm the normalized backbone output, and decoder.branches.<branch>.hidden_blocks.<index> an MLP decoder hidden-block output; the same scheme applies to BERT embedding models.

Backbone selectors contribute dim_model activations. Decoder MLP hidden-block selectors contribute their configured hidden width and receive the same flattened decoding_support * dim_model windows used during training. The default, embedding_layer_names: [backbone.final_norm], preserves the final normalised backbone representation.

BERT Model

Sequifier also supports training and inference of BERT-style masked reconstruction models.

Configuration:

• Preprocessing: Set max_target_offset: 0 for equal-width input and target windows.
• Training: Set training_objective: bert, configure bert_spec, and set decoder prediction_length equal to context_length. Enable generative and/or embedding export according to the desired inference.
• Inference: Set model_type: generative to reconstruct explicitly masked input, or model_type: embedding to output contextual representations.

Technical Details: BERT-style models use bidirectional attention and learn by reconstructing positions sampled according to bert_spec. Inference does not apply random masking; inputs that should be reconstructed must be masked explicitly, for example using mask_column during preprocessing. Embedding inference returns one contextual representation for every valid position in the input window.

Structured Ingestion

Structured ingestion allows the model to learn relationships for configured sets of input variables before they are passed to the transformer backbone.

This enables more constrained representation learning within these subspaces, and provides a structural inductive bias and may reduce parameters, depending on configuration. It can be helpful to think of them as smaller submodules that learn local structure before passing the extracted information to the transformer for longer-range dependencies.

The key modalities are self-attention, pooling, 1D, 2D and 3D convolutions, and adding learned or rotary axis embeddings.

Temporal Convolution

Separately, temporal_conv enables temporal convolutions on pass-through or embedded real or categorical variables.

Multi-Part Datasets

It is often the case that data grows and evolves, and we need the model to be updated using that data. Sequifier supports this practical reality by defining multi-part datasets as sets of data that share the same schema, categorical mappings, normalisation and storage contract, but have distinct metadata configs. In practice, this would look like processing every dataset after the first one with the metadata_config_path set to the metadata config created during the first preprocessing execution, to ensure that the properties line up as required. Also window_length, max_target_offset, normalization mode, dtypes, and file/folder storage form must match the first sequifier preprocess run.

Composable Configs

Between different training runs and hyperparameter searches, a lot of configuration can get duplicated, and it becomes hard to follow what differentiates them and where they overlap. One approach to address this is to create different config fragments, and compose them into full training and hyperparameter tuning configurations using separate 'top-level' configs, and assemble the fragments by listing them as value in additional_config_paths. Fragments can contribute disjoint nested fields, while duplicate fields, recursive fragment inclusion, and repeated files are rejected; command-line overrides are applied after composition.

Distributed Training

Sequifier supports distributed training using torch DistributedDataParallel and FullyShardedDataParallel. To make use of multi gpu support, the preprocessing step must write sharded output with merge_output: false. write_format: pt is the recommended file format; sharded parquet is also supported but currently considered beta for distributed training.

For the full guide on how to configure a distributed run, check the multi-GPU training guide.

System Requirements

Sequifier currently runs on MacOS and Ubuntu.

Citation

Please cite with:

@software{sequifier_2025,
  author = {Luithlen, Leon},
  title = {sequifier - transformers for multivariate sequence generation and representation learning},
  year = {2025},
  publisher = {GitHub},
  version = {v2.0.0.0},
  url = {https://github.com/0xideas/sequifier}
}