-
Notifications
You must be signed in to change notification settings - Fork 207
docs(custom-nodes): add V3 backend docs into versioned /custom-nodes/v3 route (V1 kept, backward compatible) #1628
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
comfyui-wiki
wants to merge
2
commits into
main
Choose a base branch
from
nav/custom-nodes-v3
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,214 @@ | ||
| --- | ||
| title: "ComfyUI Custom Node Datatypes (V3)" | ||
| description: "Reference for ComfyUI custom node data types: Python types, tensor formats (IMAGE, LATENT, MASK), custom types, and wildcards for V3 custom nodes." | ||
| --- | ||
|
|
||
| These are the most important built in datatypes. You can also [define your own](./more_on_inputs#custom-datatypes). | ||
|
|
||
| In the V3 schema every datatype is a class in the `io` module, such as `io.Image` or `io.Int`. You declare | ||
| an input with its `.Input(...)` constructor and an output with `.Output(...)`. See the | ||
| [V3 Migration guide](/custom-nodes/v3_migration) for the full mapping between the V1 type strings and the | ||
| `io` classes. | ||
|
|
||
| Datatypes are used on the client side to prevent a workflow from passing the wrong form of data into a | ||
| node - a bit like strong typing. The JavaScript client side code will generally not allow a node output to | ||
| be connected to an input of a different datatype, although a few exceptions are noted below. | ||
|
|
||
| ## Comfy datatypes | ||
|
|
||
| ### COMBO | ||
|
|
||
| `io.Combo` represents a dropdown menu widget. The value of the input is a `str`; the widget options are | ||
| provided with the `options` parameter. | ||
|
|
||
| ```python | ||
| io.Combo.Input("play_sound", options=["no", "yes"]) | ||
| ``` | ||
|
|
||
| `COMBO` inputs are often dynamically generated at run time. Because `define_schema` is a class method, the | ||
| options can be computed when the node is loaded. For instance, a checkpoint loader node might do: | ||
|
|
||
| ```python | ||
| io.Combo.Input("ckpt_name", options=folder_paths.get_filename_list("checkpoints")) | ||
| ``` | ||
|
|
||
| There is also `io.MultiCombo` for dropdowns where more than one option can be selected; its value is a | ||
| `list[str]`. | ||
|
|
||
| ### Primitive and reroute | ||
|
|
||
| Primitive and reroute nodes only exist on the client side. They do not have an intrinsic datatype, but | ||
| when connected they take on the datatype of the input or output to which they have been connected (which | ||
| is why they can't connect to a `*` input...) | ||
|
|
||
| ## Python datatypes | ||
|
|
||
| ### INT | ||
|
|
||
| `io.Int` is an integer widget input. | ||
|
|
||
| * Parameters: `default`, `min`, `max`, `step` (all optional) | ||
|
|
||
| * Python datatype: `int` | ||
|
|
||
| ```python | ||
| io.Int.Input("count", default=1, min=0, max=4096, step=1) | ||
| ``` | ||
|
|
||
| ### FLOAT | ||
|
|
||
| `io.Float` is a float widget input. | ||
|
|
||
| * Parameters: `default`, `min`, `max`, `step` (all optional) | ||
|
|
||
| * Python datatype: `float` | ||
|
|
||
| ```python | ||
| io.Float.Input("strength", default=1.0, min=0.0, max=10.0, step=0.1) | ||
| ``` | ||
|
|
||
| ### STRING | ||
|
|
||
| `io.String` is a text widget input. | ||
|
|
||
| * Parameters: `default`, `multiline`, `placeholder`, `dynamic_prompts` (all optional) | ||
|
|
||
| * Python datatype: `str` | ||
|
|
||
| ```python | ||
| io.String.Input("text", default="Hello", multiline=True) | ||
| ``` | ||
|
|
||
| ### BOOLEAN | ||
|
|
||
| `io.Boolean` is a toggle widget input. | ||
|
|
||
| * Parameters: `default`, `label_on`, `label_off` (all optional) | ||
|
|
||
| * Python datatype: `bool` | ||
|
|
||
| ```python | ||
| io.Boolean.Input("enabled", default=True, label_on="On", label_off="Off") | ||
| ``` | ||
|
|
||
| ## Tensor datatypes | ||
|
|
||
| ### IMAGE | ||
|
|
||
| * Declared with `io.Image.Input(...)` | ||
|
|
||
| * Python datatype: `torch.Tensor` with *shape* \[B,H,W,C] | ||
|
|
||
| A batch of `B` images, height `H`, width `W`, with `C` channels (generally `C=3` for `RGB`). | ||
|
|
||
| ### LATENT | ||
|
|
||
| * Declared with `io.Latent.Input(...)` | ||
|
|
||
| * Python datatype: `dict`, containing a `torch.Tensor` with *shape* \[B,C,H,W] | ||
|
|
||
| The `dict` passed contains the key `samples`, which is a `torch.Tensor` with *shape* \[B,C,H,W] representing | ||
| a batch of `B` latents, with `C` channels (generally `C=4` for existing stable diffusion models), height `H`, width `W`. | ||
|
|
||
| The height and width are 1/8 of the corresponding image size (which is the value you set in the Empty Latent Image node). | ||
|
|
||
| Other entries in the dictionary contain things like latent masks. | ||
|
|
||
| {/* TODO need to dig into this */} | ||
|
|
||
| {/* TODO new SD models might have different C values? */} | ||
|
|
||
| ### MASK | ||
|
|
||
| * Declared with `io.Mask.Input(...)` | ||
|
|
||
| * Python datatype: `torch.Tensor` with *shape* \[H,W] or \[B,C,H,W] | ||
|
|
||
| ### AUDIO | ||
|
|
||
| * Declared with `io.Audio.Input(...)` | ||
|
|
||
| * Python datatype: `dict`, containing a `torch.Tensor` with *shape* \[B, C, T] and a sample rate. | ||
|
|
||
| The `dict` passed contains the key `waveform`, which is a `torch.Tensor` with *shape* \[B, C, T] representing a batch of `B` audio samples, with `C` channels (`C=2` for stereo and `C=1` for mono), and `T` time steps (i.e., the number of audio samples). | ||
|
|
||
| The `dict` contains another key `sample_rate`, which indicates the sampling rate of the audio. | ||
|
|
||
| ## Custom Sampling datatypes | ||
|
|
||
| ### Noise | ||
|
|
||
| The `NOISE` datatype represents a *source* of noise (not the actual noise itself). It can be represented by any Python object | ||
| that provides a method to generate noise, with the signature `generate_noise(self, input_latent:Tensor) -> Tensor`, and a | ||
| property, `seed:Optional[int]`. | ||
|
|
||
| <Tip>The `seed` is passed into `sample` guider in the `SamplerCustomAdvanced`, but does not appear to be used in any of the standard guiders. | ||
| It is Optional, so you can generally set it to None.</Tip> | ||
|
|
||
| When noise is to be added, the latent is passed into this method, which should return a `Tensor` of the same shape containing the noise. | ||
|
|
||
| See the [noise mixing example](/custom-nodes/backend/snippets#creating-noise-variations) | ||
|
|
||
| ### Sampler | ||
|
|
||
| The `SAMPLER` datatype represents a sampler, which is represented as a Python object providing a `sample` method. | ||
| Stable diffusion sampling is beyond the scope of this guide; see `comfy/samplers.py` if you want to dig into this part of the code. | ||
|
|
||
| ### Sigmas | ||
|
|
||
| The `SIGMAS` datatypes represents the values of sigma before and after each step in the sampling process, as produced by a scheduler. | ||
| This is represented as a one-dimensional tensor, of length `steps+1`, where each element represents the noise expected to be present | ||
| before the corresponding step, with the final value representing the noise present after the final step. | ||
|
|
||
| A `normal` scheduler, with 20 steps and denoise of 1, for an SDXL model, produces: | ||
|
|
||
| ``` | ||
| tensor([14.6146, 10.7468, 8.0815, 6.2049, 4.8557, | ||
| 3.8654, 3.1238, 2.5572, 2.1157, 1.7648, | ||
| 1.4806, 1.2458, 1.0481, 0.8784, 0.7297, | ||
| 0.5964, 0.4736, 0.3555, 0.2322, 0.0292, 0.0000]) | ||
| ``` | ||
|
|
||
| <Tip>The starting value of sigma depends on the model, which is why a scheduler node requires a `MODEL` input to produce a SIGMAS output</Tip> | ||
|
|
||
| ### Guider | ||
|
|
||
| A `GUIDER` is a generalisation of the denoising process, as 'guided' by a prompt or any other form of conditioning. In Comfy the guider is | ||
| represented by a `callable` Python object providing a `__call__(*args, **kwargs)` method which is called by the sample. | ||
|
|
||
| The `__call__` method takes (in `args[0]`) a batch of noisy latents (tensor `[B,C,H,W]`), and returns a prediction of the noise (a tensor of the same shape). | ||
|
|
||
| ## Model datatypes | ||
|
|
||
| There are a number of more technical datatypes for stable diffusion models. The most significant ones are `io.Model`, `io.CLIP`, | ||
| `io.VAE` and `io.Conditioning`. | ||
| Working with these is (for the time being) beyond the scope of this guide! {/* TODO but maybe not forever */} | ||
|
|
||
| ## Additional Parameters | ||
|
|
||
| The `Input` constructors of the widget datatypes accept a number of optional parameters that configure the | ||
| widget. Below is a list of the officially supported parameters. (The legacy V1 schema spelled these as keys | ||
| in the input options dictionary, e.g. `forceInput` and `rawLink`; in V3 they are snake_case parameters of | ||
| the `Input` constructors.) | ||
|
|
||
| <Warning>You can use additional keys for your own custom widgets, but should *not* reuse any of the parameters below for other purposes.</Warning> | ||
|
|
||
| | Parameter | Applies to | Description | | ||
| | --------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | | ||
| | `default` | widget inputs | The default value of the widget | | ||
| | `min` | `io.Int`, `io.Float` | The minimum value of the widget | | ||
| | `max` | `io.Int`, `io.Float` | The maximum value of the widget | | ||
| | `step` | `io.Int`, `io.Float` | The amount to increment or decrement a widget | | ||
| | `multiline` | `io.String` | Use a multiline text box | | ||
| | `placeholder` | `io.String` | Placeholder text to display in the UI when empty | | ||
| | `dynamic_prompts` | `io.String` | Causes the front-end to evaluate dynamic prompts | | ||
| | `options` | `io.Combo`, `io.MultiCombo` | The dropdown options | | ||
| | `label_on`, `label_off` | `io.Boolean` | The labels to use in the UI when the bool is `True` / `False` | | ||
| | `control_after_generate` | `io.Int`, `io.Combo` | Adds a "control after generate" widget (pass `True`, or an `io.ControlAfterGenerate` enum value) | | ||
| | `socketless` | widget inputs | Hide the input socket, so the widget cannot be connected to an upstream node | | ||
| | `force_input` | widget inputs | Display the input as a socket instead of a widget, and don't allow converting it back to a widget | | ||
| | `optional` | all inputs | Declares the input optional, so it does not have to be connected | | ||
| | `lazy` | all inputs | Declares that this input uses [Lazy Evaluation](./lazy_evaluation) | | ||
| | `raw_link` | all inputs | When a link exists, rather than receiving the evaluated value, you will receive the link (i.e. `["nodeId", <outputIndex>]`). Primarily useful when your node uses [Node Expansion](/custom-nodes/backend/expansion). | | ||
| | `tooltip` | all inputs | Tooltip text to display when hovering over the input | | ||
| | `advanced` | all inputs | Hides the input behind an "Advanced" toggle in the UI | | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
Repository: Comfy-Org/docs
Length of output: 1330
🤖 get_repo_knowledge executed:
get_repo_knowledge Comfy-Org/docs /tmp/coderabbit-repo-knowledge/comfy-org-docs-ba19da15/learningsLength of output: 1579
🏁 Script executed:
Repository: Comfy-Org/docs
Length of output: 45279
🌐 Web query:
ComfyUI V3 io.Mask.Input tensor shape [B,H,W] [H,W] runtime contract💡 Result:
In ComfyUI, the standard runtime contract for a MASK tensor is [B, H, W] [1][2]. While historical or internal representations might use [H, W] (with an implicit batch dimension of 1) [1] or occasionally [B, 1, H, W] [3][4], custom nodes should ideally expect and produce masks in the [B, H, W] format to maintain compatibility with the broader ComfyUI ecosystem [3]. Key details regarding the MASK tensor contract: Standard Shape: The canonical format is [B, H, W], where B is the batch size, H is height, and W is width [1][2]. Implicit Dimensions: It is common to encounter tensors with shape [H, W] in contexts where the batch dimension has been squeezed [1]. Conversely, some processes (or legacy/alternative implementations) may use a 4D shape [B, 1, H, W] [3][4]. Compatibility Issues: Because many downstream nodes contain strict validation checks for shape (e.g., asserting that an input has exactly 3 dimensions), nodes that return 4D tensors ([B, 1, H, W]) may cause runtime errors in standard ComfyUI workflows [3]. Best Practices for Custom Nodes: If you are developing a custom node, it is recommended to normalize the mask output to [B, H, W] [3]. If a node receives a mask input, robust implementations should ideally handle or validate the input shape, such as explicitly squeezing singleton dimensions if necessary, to ensure compatibility with various mask producers [3]. The ComfyUI documentation explicitly notes that while libraries like numpy or PIL use 2D arrays [H, W], ComfyUI mask tensors follow the [B, H, W] convention [1]. [1][3][4]
Citations:
🏁 Script executed:
Repository: Comfy-Org/docs
Length of output: 2190
Document the canonical
MASKshape.The canonical shape is
[B,H,W].[H,W]is only a batch-squeezed form.[B,C,H,W]is not theMASKshape and confuses masks with image or latent tensors. Update the line to document[B,H,W]; keep the shape contract precise, or the dimensions may run amask.🤖 Prompt for AI Agents