diff --git a/custom-nodes/intro.mdx b/custom-nodes/intro.mdx index 7f36c9b92..b990c51ae 100644 --- a/custom-nodes/intro.mdx +++ b/custom-nodes/intro.mdx @@ -22,6 +22,6 @@ Custom Nodes extend ComfyUI with new server-side behavior, user-interface featur ## Choose a starting point - **New to custom nodes:** start with [Custom Nodes Overview](/custom-nodes/overview). -- **Building the backend:** see [Python (Backend)](/custom-nodes/backend/server_overview). +- **Building the backend:** see [Python (Backend) · V3](/custom-nodes/v3/backend/server_overview). New nodes should use the V3 schema; find the [legacy V1 schema](/custom-nodes/backend/server_overview) in the V1 section. - **Extending the UI:** see [JavaScript (UI)](/custom-nodes/js/javascript_overview). - **Publishing a package:** start with the [Registry Overview](/registry/overview). diff --git a/custom-nodes/overview.mdx b/custom-nodes/overview.mdx index 3039a23e3..914ba8c29 100644 --- a/custom-nodes/overview.mdx +++ b/custom-nodes/overview.mdx @@ -54,8 +54,8 @@ In a small number of cases, the UI features and the server need to interact with Build and register a working custom node, step by step. - - The node class contract: properties, inputs, return types. + + The V3 node class contract: properties, inputs, return types. Extend the ComfyUI frontend from your node package. diff --git a/custom-nodes/v3/backend/datatypes.mdx b/custom-nodes/v3/backend/datatypes.mdx new file mode 100644 index 000000000..255ce9c34 --- /dev/null +++ b/custom-nodes/v3/backend/datatypes.mdx @@ -0,0 +1,214 @@ +--- +title: "ComfyUI Custom Node Datatypes (V3)" +description: "Reference for ComfyUI custom node data types: Python types, tensor formats (IMAGE, LATENT, MASK), custom types, and wildcards for V3 custom nodes." +--- + +These are the most important built in datatypes. You can also [define your own](./more_on_inputs#custom-datatypes). + +In the V3 schema every datatype is a class in the `io` module, such as `io.Image` or `io.Int`. You declare +an input with its `.Input(...)` constructor and an output with `.Output(...)`. See the +[V3 Migration guide](/custom-nodes/v3_migration) for the full mapping between the V1 type strings and the +`io` classes. + +Datatypes are used on the client side to prevent a workflow from passing the wrong form of data into a +node - a bit like strong typing. The JavaScript client side code will generally not allow a node output to +be connected to an input of a different datatype, although a few exceptions are noted below. + +## Comfy datatypes + +### COMBO + +`io.Combo` represents a dropdown menu widget. The value of the input is a `str`; the widget options are +provided with the `options` parameter. + +```python +io.Combo.Input("play_sound", options=["no", "yes"]) +``` + +`COMBO` inputs are often dynamically generated at run time. Because `define_schema` is a class method, the +options can be computed when the node is loaded. For instance, a checkpoint loader node might do: + +```python +io.Combo.Input("ckpt_name", options=folder_paths.get_filename_list("checkpoints")) +``` + +There is also `io.MultiCombo` for dropdowns where more than one option can be selected; its value is a +`list[str]`. + +### Primitive and reroute + +Primitive and reroute nodes only exist on the client side. They do not have an intrinsic datatype, but +when connected they take on the datatype of the input or output to which they have been connected (which +is why they can't connect to a `*` input...) + +## Python datatypes + +### INT + +`io.Int` is an integer widget input. + +* Parameters: `default`, `min`, `max`, `step` (all optional) + +* Python datatype: `int` + +```python +io.Int.Input("count", default=1, min=0, max=4096, step=1) +``` + +### FLOAT + +`io.Float` is a float widget input. + +* Parameters: `default`, `min`, `max`, `step` (all optional) + +* Python datatype: `float` + +```python +io.Float.Input("strength", default=1.0, min=0.0, max=10.0, step=0.1) +``` + +### STRING + +`io.String` is a text widget input. + +* Parameters: `default`, `multiline`, `placeholder`, `dynamic_prompts` (all optional) + +* Python datatype: `str` + +```python +io.String.Input("text", default="Hello", multiline=True) +``` + +### BOOLEAN + +`io.Boolean` is a toggle widget input. + +* Parameters: `default`, `label_on`, `label_off` (all optional) + +* Python datatype: `bool` + +```python +io.Boolean.Input("enabled", default=True, label_on="On", label_off="Off") +``` + +## Tensor datatypes + +### IMAGE + +* Declared with `io.Image.Input(...)` + +* Python datatype: `torch.Tensor` with *shape* \[B,H,W,C] + +A batch of `B` images, height `H`, width `W`, with `C` channels (generally `C=3` for `RGB`). + +### LATENT + +* Declared with `io.Latent.Input(...)` + +* Python datatype: `dict`, containing a `torch.Tensor` with *shape* \[B,C,H,W] + +The `dict` passed contains the key `samples`, which is a `torch.Tensor` with *shape* \[B,C,H,W] representing +a batch of `B` latents, with `C` channels (generally `C=4` for existing stable diffusion models), height `H`, width `W`. + +The height and width are 1/8 of the corresponding image size (which is the value you set in the Empty Latent Image node). + +Other entries in the dictionary contain things like latent masks. + +{/* TODO need to dig into this */} + +{/* TODO new SD models might have different C values? */} + +### MASK + +* Declared with `io.Mask.Input(...)` + +* Python datatype: `torch.Tensor` with *shape* \[H,W] or \[B,C,H,W] + +### AUDIO + +* Declared with `io.Audio.Input(...)` + +* Python datatype: `dict`, containing a `torch.Tensor` with *shape* \[B, C, T] and a sample rate. + +The `dict` passed contains the key `waveform`, which is a `torch.Tensor` with *shape* \[B, C, T] representing a batch of `B` audio samples, with `C` channels (`C=2` for stereo and `C=1` for mono), and `T` time steps (i.e., the number of audio samples). + +The `dict` contains another key `sample_rate`, which indicates the sampling rate of the audio. + +## Custom Sampling datatypes + +### Noise + +The `NOISE` datatype represents a *source* of noise (not the actual noise itself). It can be represented by any Python object +that provides a method to generate noise, with the signature `generate_noise(self, input_latent:Tensor) -> Tensor`, and a +property, `seed:Optional[int]`. + +The `seed` is passed into `sample` guider in the `SamplerCustomAdvanced`, but does not appear to be used in any of the standard guiders. +It is Optional, so you can generally set it to None. + +When noise is to be added, the latent is passed into this method, which should return a `Tensor` of the same shape containing the noise. + +See the [noise mixing example](/custom-nodes/backend/snippets#creating-noise-variations) + +### Sampler + +The `SAMPLER` datatype represents a sampler, which is represented as a Python object providing a `sample` method. +Stable diffusion sampling is beyond the scope of this guide; see `comfy/samplers.py` if you want to dig into this part of the code. + +### Sigmas + +The `SIGMAS` datatypes represents the values of sigma before and after each step in the sampling process, as produced by a scheduler. +This is represented as a one-dimensional tensor, of length `steps+1`, where each element represents the noise expected to be present +before the corresponding step, with the final value representing the noise present after the final step. + +A `normal` scheduler, with 20 steps and denoise of 1, for an SDXL model, produces: + +``` +tensor([14.6146, 10.7468, 8.0815, 6.2049, 4.8557, + 3.8654, 3.1238, 2.5572, 2.1157, 1.7648, + 1.4806, 1.2458, 1.0481, 0.8784, 0.7297, + 0.5964, 0.4736, 0.3555, 0.2322, 0.0292, 0.0000]) +``` + +The starting value of sigma depends on the model, which is why a scheduler node requires a `MODEL` input to produce a SIGMAS output + +### Guider + +A `GUIDER` is a generalisation of the denoising process, as 'guided' by a prompt or any other form of conditioning. In Comfy the guider is +represented by a `callable` Python object providing a `__call__(*args, **kwargs)` method which is called by the sample. + +The `__call__` method takes (in `args[0]`) a batch of noisy latents (tensor `[B,C,H,W]`), and returns a prediction of the noise (a tensor of the same shape). + +## Model datatypes + +There are a number of more technical datatypes for stable diffusion models. The most significant ones are `io.Model`, `io.CLIP`, +`io.VAE` and `io.Conditioning`. +Working with these is (for the time being) beyond the scope of this guide! {/* TODO but maybe not forever */} + +## Additional Parameters + +The `Input` constructors of the widget datatypes accept a number of optional parameters that configure the +widget. Below is a list of the officially supported parameters. (The legacy V1 schema spelled these as keys +in the input options dictionary, e.g. `forceInput` and `rawLink`; in V3 they are snake_case parameters of +the `Input` constructors.) + +You can use additional keys for your own custom widgets, but should *not* reuse any of the parameters below for other purposes. + +| Parameter | Applies to | Description | +| --------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------- | +| `default` | widget inputs | The default value of the widget | +| `min` | `io.Int`, `io.Float` | The minimum value of the widget | +| `max` | `io.Int`, `io.Float` | The maximum value of the widget | +| `step` | `io.Int`, `io.Float` | The amount to increment or decrement a widget | +| `multiline` | `io.String` | Use a multiline text box | +| `placeholder` | `io.String` | Placeholder text to display in the UI when empty | +| `dynamic_prompts` | `io.String` | Causes the front-end to evaluate dynamic prompts | +| `options` | `io.Combo`, `io.MultiCombo` | The dropdown options | +| `label_on`, `label_off` | `io.Boolean` | The labels to use in the UI when the bool is `True` / `False` | +| `control_after_generate` | `io.Int`, `io.Combo` | Adds a "control after generate" widget (pass `True`, or an `io.ControlAfterGenerate` enum value) | +| `socketless` | widget inputs | Hide the input socket, so the widget cannot be connected to an upstream node | +| `force_input` | widget inputs | Display the input as a socket instead of a widget, and don't allow converting it back to a widget | +| `optional` | all inputs | Declares the input optional, so it does not have to be connected | +| `lazy` | all inputs | Declares that this input uses [Lazy Evaluation](./lazy_evaluation) | +| `raw_link` | all inputs | When a link exists, rather than receiving the evaluated value, you will receive the link (i.e. `["nodeId", ]`). Primarily useful when your node uses [Node Expansion](/custom-nodes/backend/expansion). | +| `tooltip` | all inputs | Tooltip text to display when hovering over the input | +| `advanced` | all inputs | Hides the input behind an "Advanced" toggle in the UI | diff --git a/custom-nodes/v3/backend/lazy_evaluation.mdx b/custom-nodes/v3/backend/lazy_evaluation.mdx new file mode 100644 index 000000000..9852ba673 --- /dev/null +++ b/custom-nodes/v3/backend/lazy_evaluation.mdx @@ -0,0 +1,184 @@ +--- +title: "Lazy Evaluation (V3)" +description: "Learn how lazy evaluation works in ComfyUI (v0.2.0+). Discover how to defer input evaluation, optimize VRAM usage, and implement lazy inputs in V3 custom nodes." +--- + +## Lazy Evaluation + +By default, all inputs are evaluated before a node can be run. Sometimes, however, an input won't +necessarily be used and evaluating it would result in unnecessary processing. Here are some examples of +nodes where lazy evaluation may be beneficial: + +1. A `ModelMergeSimple` node where the ratio is either `0.0` (in which case the first model doesn't need to be loaded) or `1.0` (in which case the second model doesn't need to be loaded). +2. Interpolation between two images where the ratio (or mask) is either entirely `0.0` or entirely `1.0`. +3. A Switch node where one input determines which of the other inputs will be passed through. + +There is very little cost in making an input lazy. If it's something you can do, you generally should. + +### Creating Lazy Inputs + +There are two steps to making an input a "lazy" input. They are: + +1. Mark the input as lazy in the schema, by passing `lazy=True` to its `Input` definition +2. Define a class method named `check_lazy_status` that will be called prior to evaluation to determine if any more inputs are necessary. + +To demonstrate these, we'll make a "MixImages" node that interpolates between two images according to a +mask. If the entire mask is `0.0`, we don't need to evaluate any part of the tree leading up to the second +image. If the entire mask is `1.0`, we can skip evaluating the first image. + +#### Defining the schema + +Declaring that an input is lazy is as simple as passing `lazy=True` to the input's definition. + +```python +from comfy_api.latest import io + + +class LazyMixImages(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="LazyMixImages", + display_name="Lazy Mix Images", + category="examples", + inputs=[ + io.Image.Input("image1", lazy=True), + io.Image.Input("image2", lazy=True), + io.Mask.Input("mask"), + ], + outputs=[ + io.Image.Output(), + ], + ) +``` + +In this example, `image1` and `image2` are both marked as lazy inputs, but `mask` will always be evaluated. + +#### Defining `check_lazy_status` + +A `check_lazy_status` method is called if there are one or more lazy inputs that are not yet available. It +receives the same arguments as `execute`. All available inputs are passed in with their final values while +unavailable lazy inputs have a value of `None`. + +When a lazy input was defined with `INPUT_IS_LIST = True`, an unevaluated input is passed to +`check_lazy_status` as `(None,)` rather than `None`, so an `is None` check would miss it. Instead, check for +the `(None,)` sentinel to ensure required inputs are not omitted. + +The responsibility of `check_lazy_status` is to return a list of the names of any lazy inputs that are +needed to proceed. If all lazy inputs are available, the function should return an empty list. + +Note that `check_lazy_status` may be called multiple times. (For example, you might find after evaluating +one lazy input that you need to evaluate another.) + +In V3 `check_lazy_status` is a class method, like `execute`. (In the legacy V1 schema it was a plain method.) + +```python +@classmethod +def check_lazy_status(cls, mask, image1=None, image2=None): + mask_min = float(mask.min()) + mask_max = float(mask.max()) + needed = [] + if image1 is None and not (mask_min == 1.0 and mask_max == 1.0): + needed.append("image1") + if image2 is None and not (mask_min == 0.0 and mask_max == 0.0): + needed.append("image2") + return needed + +@classmethod +def execute(cls, mask, image1=None, image2=None) -> io.NodeOutput: + mask_min = float(mask.min()) + mask_max = float(mask.max()) + if mask_min == 0.0 and mask_max == 0.0: + return io.NodeOutput(image1) + if mask_min == 1.0 and mask_max == 1.0: + return io.NodeOutput(image2) + # Not trying to handle different batch sizes here just to keep the demo simple + return io.NodeOutput(image1 * (1.0 - mask) + image2 * mask) +``` + +### Full Example + +```python +from comfy_api.latest import ComfyExtension, io + + +class LazyMixImages(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="LazyMixImages", + display_name="Lazy Mix Images", + category="examples", + inputs=[ + io.Image.Input("image1", lazy=True), + io.Image.Input("image2", lazy=True), + io.Mask.Input("mask"), + ], + outputs=[ + io.Image.Output(), + ], + ) + + @classmethod + def check_lazy_status(cls, mask, image1=None, image2=None): + mask_min = float(mask.min()) + mask_max = float(mask.max()) + needed = [] + if image1 is None and not (mask_min == 1.0 and mask_max == 1.0): + needed.append("image1") + if image2 is None and not (mask_min == 0.0 and mask_max == 0.0): + needed.append("image2") + return needed + + @classmethod + def execute(cls, mask, image1=None, image2=None) -> io.NodeOutput: + mask_min = float(mask.min()) + mask_max = float(mask.max()) + if mask_min == 0.0 and mask_max == 0.0: + return io.NodeOutput(image1) + if mask_min == 1.0 and mask_max == 1.0: + return io.NodeOutput(image2) + # Not trying to handle different batch sizes here just to keep the demo simple + return io.NodeOutput(image1 * (1.0 - mask) + image2 * mask) + + +class ExampleExtension(ComfyExtension): + async def get_node_list(self) -> list[type[io.ComfyNode]]: + return [LazyMixImages] + + +async def comfy_entrypoint() -> ExampleExtension: + return ExampleExtension() +``` + +## Execution Blocking + +While Lazy Evaluation is the recommended way to "disable" part of a graph, there are times when you want to +disable a node that doesn't implement lazy evaluation itself. If it's an output node that you developed +yourself, you should just add lazy evaluation as follows: + +1. Add a required (if this is a new node) or optional (if you care about backward compatibility) input for `enabled` that defaults to `True` +2. Make all other inputs lazy inputs +3. Only evaluate the other inputs if `enabled` is `True` + +To block execution from a node whose output may be invalid or meaningless, return an +`io.NodeOutput` with a `block_execution` message. Comfy replaces every output of the node with an +`ExecutionBlocker` carrying that message. Any nodes which receive an `ExecutionBlocker` as input will skip +execution and return that `ExecutionBlocker` for any outputs, so the message is reported to the user when a +blocked output is actually used. + +```python +@classmethod +def execute(cls, ckpt_name) -> io.NodeOutput: + ckpt_path = folder_paths.get_full_path("checkpoints", ckpt_name) + model, clip, vae = load_checkpoint(ckpt_path) + if vae is None: + # This error is more useful than a "'NoneType' has no attribute" error + # in a later node + return io.NodeOutput( + block_execution=f"No VAE contained in the loaded model {ckpt_name}" + ) + return io.NodeOutput(model, clip, vae) +``` + +**There is intentionally no way to stop an ExecutionBlocker from propagating forward.** If you think you want this, you should really be using Lazy Evaluation. diff --git a/custom-nodes/v3/backend/lifecycle.mdx b/custom-nodes/v3/backend/lifecycle.mdx new file mode 100644 index 000000000..fb9a108d4 --- /dev/null +++ b/custom-nodes/v3/backend/lifecycle.mdx @@ -0,0 +1,75 @@ +--- +title: "Lifecycle (V3)" +description: "Understand how ComfyUI loads custom nodes: the comfy_entrypoint function and ComfyExtension class that register V3 nodes, plus WEB_DIRECTORY for deploying client-side JavaScript." +--- + +## How Comfy loads custom nodes + +When Comfy starts, it scans the directory `custom_nodes` for Python modules, and attempts to load them. +A module is treated as an extension when it exports a `comfy_entrypoint` function (the V3 schema) or a +`NODE_CLASS_MAPPINGS` dictionary (the legacy V1 schema). See the +[V3 Migration guide](/custom-nodes/v3_migration) for the differences between the two. + +A custom-node package is a directory containing an `__init__.py` file. +`__all__` affects wildcard imports only and does not control custom-node discovery. + +### __init__.py + +`__init__.py` is executed when Comfy attempts to import the module. If the import succeeds, Comfy calls +the module's `comfy_entrypoint` function, which returns a `ComfyExtension` instance. The extension's +`get_node_list` method provides the node classes defined by the module, and those nodes become available +in Comfy. If there is an error in your code, Comfy will continue, but will report the module as having +failed to load. So check the Python console! + +A very simple `__init__.py` file would look like this: + +```python +from comfy_api.latest import ComfyExtension + +from .python_file import MyCustomNode + + +class MyExtension(ComfyExtension): + async def get_node_list(self) -> list[type[MyCustomNode]]: + return [MyCustomNode] + + +async def comfy_entrypoint() -> MyExtension: + return MyExtension() +``` + +#### comfy_entrypoint + +`comfy_entrypoint` is the function Comfy calls to discover the nodes in your module. It may be declared +`async` or not, but it must return an instance of `ComfyExtension`. The `get_node_list` method on the +extension must be `async`, and returns the list of node classes the extension provides. + +The `node_id` of each node (its unique name across the Comfy install) and its display name, category and +other properties are defined in the node class's `define_schema` method, rather than in the +`__init__.py`. See [Properties](./server_overview) for the schema fields. + +#### NODE_CLASS_MAPPINGS (legacy V1) + +Modules written against the legacy V1 schema can still export `NODE_CLASS_MAPPINGS`, a `dict` mapping +each custom node name (unique across the Comfy install) to its node class. Comfy loads those modules +without calling `comfy_entrypoint`. + +```python +from .python_file import MyCustomNode +NODE_CLASS_MAPPINGS = {"MyCustomNode": MyCustomNode} +``` + +`NODE_DISPLAY_NAME_MAPPINGS` was the legacy way to give a node a display name different from its unique +name. In V3 this is done with the `display_name` field of the schema. + +#### WEB_DIRECTORY + +If you are deploying client side code, you will also need to export the path, relative to the module, in +which the JavaScript files are to be found. It is conventional to place these in a subdirectory of your +custom node named `js`. Comfy also registers a web directory automatically when your `pyproject.toml` +contains a `[tool.comfy]` section with a `web` key pointing at one. + +*Only* `.js` files will be served; you can't deploy `.css` or other types in this way + +In previous versions of Comfy, `__init__.py` was required to copy the JavaScript files into the main Comfy web +subdirectory. You will still see code that does this. Don't. diff --git a/custom-nodes/v3/backend/lists.mdx b/custom-nodes/v3/backend/lists.mdx new file mode 100644 index 000000000..cebc5a097 --- /dev/null +++ b/custom-nodes/v3/backend/lists.mdx @@ -0,0 +1,83 @@ +--- +title: "Data lists (V3)" +description: "Learn how ComfyUI handles data as Python lists internally, including length-one processing, list processing for batches, and how to use the V3 is_input_list and is_output_list schema options when developing custom nodes." +--- + +## Length one processing + +Internally, the Comfy server represents data flowing from one node to the next as a Python `list`, normally +length 1, of the relevant datatype. In normal operation, when a node returns an output, each element in the +output is separately wrapped in a list (length 1); then when the next node is called, the data is unwrapped +and passed to the `execute` method. + +You generally don't need to worry about this, since Comfy does the wrapping and unwrapping. + +This isn't about batches. A batch (of, for instance, latents, or images) is a *single entry* in the list (see [tensor datatypes](/custom-nodes/backend/images_and_masks)) + +## List processing + +In some circumstance, multiple data instances are processed in a single workflow, in which case the internal +data will be a list containing the data instances. An example of this might be processing a series of images +one at a time to avoid running out of VRAM, or handling images of different sizes. + +By default, Comfy will process the values in the list sequentially: + +- if the inputs are `list`s of different lengths, the shorter ones are padded by repeating the last value +- the `execute` method is called once for each value in the input lists +- the outputs are `list`s, each of which is the same length as the longest input + +The relevant code can be found in the method `map_node_over_list` in `execution.py`. + +However, as Comfy wraps node outputs into a `list` of length one, if the values returned by a custom node +contain a `list`, that `list` will be wrapped, and treated as a single piece of data. + +Two V3 schema options change this behaviour: + +- `is_input_list=True` on the schema: the node receives the *whole* list in a single call, instead of being + called once per item. All inputs become `list[type]`, regardless of how many items are passed in. This + replaces the legacy V1 class attribute `INPUT_IS_LIST`. +- `is_output_list=True` on an output: the list returned for that output is not wrapped, and is treated as a + series of data for sequential processing by downstream nodes. This replaces the legacy V1 class attribute + `OUTPUT_IS_LIST`. + +To show how the two options work together, here's an `ImageRebatch`-style node written in the V3 schema. It +takes one or more batches of images (received as a list, because `is_input_list=True`) and rebatches them +into batches of the requested size: + +`is_input_list` is node level - all inputs get the same treatment. So the value of the `batch_size` widget is given by `batch_size[0]`. + +```python +from comfy_api.latest import ComfyExtension, io + + +class ImageRebatch(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="ImageRebatch", + category="image/batch", + inputs=[ + io.Image.Input("images"), + io.Int.Input("batch_size", default=1, min=1, max=4096), + ], + outputs=[ + io.Image.Output(is_output_list=True), + ], + is_input_list=True, + ) + + @classmethod + def execute(cls, images, batch_size) -> io.NodeOutput: + batch_size = batch_size[0] # everything comes as a list, so batch_size is list[int] + + output_list = [] + all_images = [] + for img in images: # each img is a batch of images + for i in range(img.shape[0]): # each i is a single image + all_images.append(img[i:i+1]) + + for i in range(0, len(all_images), batch_size): # take batch_size chunks and turn each into a new batch + output_list.append(torch.cat(all_images[i:i+batch_size], dim=0)) # will die horribly if the image batches had different width or height! + + return io.NodeOutput(output_list) +``` diff --git a/custom-nodes/v3/backend/more_on_inputs.mdx b/custom-nodes/v3/backend/more_on_inputs.mdx new file mode 100644 index 000000000..357706960 --- /dev/null +++ b/custom-nodes/v3/backend/more_on_inputs.mdx @@ -0,0 +1,152 @@ +--- +title: "Hidden and Flexible inputs (V3)" +description: "ComfyUI hidden inputs (UNIQUE_ID, PROMPT, EXTRA_PNGINFO, DYNPROMPT) and flexible inputs, including custom datatypes and wildcard inputs for custom nodes (V3 schema)." +--- + +## Hidden inputs + +Alongside the inputs declared in the schema, which create corresponding inputs or widgets on the +client-side, custom nodes can request certain information from the server through *hidden* inputs. Hidden +inputs are not visible in the UI. + +In the V3 schema, hidden inputs are requested by passing a list of `io.Hidden` enum values to the `hidden` +parameter of the schema. During execution, the values are available on the node class as `cls.hidden`, so +an `execute` method can read them by name: + +```python +from comfy_api.latest import io + +class MyNode(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="MyNode", + inputs=[io.String.Input("text")], + outputs=[io.String.Output()], + hidden=[ + io.Hidden.unique_id, + io.Hidden.prompt, + io.Hidden.extra_pnginfo, + ], + ) + + @classmethod + def execute(cls, text) -> io.NodeOutput: + # hidden values are accessed via cls.hidden + print(cls.hidden.unique_id) + print(cls.hidden.prompt) + print(cls.hidden.extra_pnginfo) + return io.NodeOutput(text) +``` + +Available hidden values (see the [V3 Migration guide](/custom-nodes/v3_migration) for the full list): + +### UNIQUE_ID + +`io.Hidden.unique_id` provides the unique identifier of the node, and matches the `id` property of the +node on the client side. It is commonly used in client-server communications (see +[messages](/development/comfyui-server/comms_messages#getting-node-id)). + +### PROMPT + +`io.Hidden.prompt` provides the complete prompt sent by the client to the server. See +[the prompt object](/custom-nodes/js/javascript_objects_and_hijacking#prompt) for a full description. + +### EXTRA_PNGINFO + +`io.Hidden.extra_pnginfo` provides a dictionary that will be copied into the metadata of any `.png` files +saved. Custom nodes can store additional information in this dictionary for saving (or as a way to +communicate with a downstream node). + +Note that if Comfy is started with the `disable_metadata` option, this data won't be saved. + +### DYNPROMPT + +`io.Hidden.dynprompt` provides an instance of `comfy_execution.graph.DynamicPrompt`. It differs from +`PROMPT` in that it may mutate during the course of execution in response to [Node Expansion](/custom-nodes/backend/expansion). + +`DYNPROMPT` should only be used for advanced cases (like implementing loops in custom nodes). + +## Flexible inputs + +### Custom datatypes + +If you want to pass data between your own custom nodes, you may find it helpful to define a custom +datatype. This is (almost) as simple as just choosing a name for the datatype, which should be a unique +string in upper case, such as `CHEESE`. + +Create the type with the `io.Custom` helper (or the `@io.comfytype` decorator for a full class +definition), then use it for inputs and outputs. The Comfy client will only allow `CHEESE` outputs to +connect to a `CHEESE` input. A `CHEESE` value can be any Python object. + +```python +from comfy_api.latest import io + +# Create the custom type once, then reuse it +Cheese = io.Custom("CHEESE") + +class CheeseNode(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="CheeseNode", + inputs=[Cheese.Input("my_cheese")], + outputs=[Cheese.Output()], + ) +``` + +Because the Comfy client doesn't know anything about `CHEESE`, it can't display a widget for it. Custom +datatype inputs are therefore socket inputs: they must be connected to an upstream node, and the widget +conversion options (`force_input` and `socketless`) don't apply to them. If you want a widget fallback for +your custom type, you need to define a custom widget for it, which is a topic for another day. + +### Wildcard inputs + +The frontend allows `*` to indicate that an input can be connected to any source. In the V3 schema, +`io.AnyType` provides this wildcard type directly: + +```python +class AnyNode(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="AnyNode", + inputs=[ + io.AnyType.Input("anything"), + ], + outputs=[io.AnyType.Output()], + ) +``` + +With `io.AnyType`, type validation accepts any input, so the legacy V1 workaround (adding an +`input_types` argument to `VALIDATE_INPUTS` in order to skip backend validation) is not needed. It's up to +the node to make sense of the data that is passed. + +### Dynamically created inputs + +If inputs are dynamically created on the client side, they can't be defined in the Python source code. + +The V3 schema handles this with the `accept_all_inputs` flag. When it is `True`, all inputs from the +prompt that are not defined in the schema are passed through to `execute` as keyword arguments: + +```python +class FlexibleNode(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="FlexibleNode", + inputs=[ + # inputs that are always present can still be declared here + ], + outputs=[io.Image.Output()], + accept_all_inputs=True, + ) + + @classmethod + def execute(cls, **kwargs) -> io.NodeOutput: + # the dynamically created input data will be in the keyword arguments + ... +``` + +Note that inputs declared in the schema are still validated and dispatched normally; `accept_all_inputs` +only affects the inputs that are *not* declared. diff --git a/custom-nodes/v3/backend/server_overview.mdx b/custom-nodes/v3/backend/server_overview.mdx new file mode 100644 index 000000000..857cf5474 --- /dev/null +++ b/custom-nodes/v3/backend/server_overview.mdx @@ -0,0 +1,220 @@ +--- +title: "Properties (V3)" +description: "Properties of a custom node (V3 schema)." +--- + + +The node definitions on this page use the modern V3 schema (`io.ComfyNode` with `define_schema` and a `comfy_entrypoint` function), which is how the bundled [`custom_nodes/example_node.py.example`](https://github.com/comfyanonymous/ComfyUI/blob/master/custom_nodes/example_node.py.example) is written. The legacy V1 schema (`INPUT_TYPES` / `RETURN_TYPES`) is still fully supported. For the differences and how to migrate, see the [V3 Migration guide](/custom-nodes/v3_migration). + + +### Simple Example + +Here's the code for the Invert Image Node, which gives an overview of the key concepts in custom node development. + +```python +from comfy_api.latest import ComfyExtension, io + +class InvertImageNode(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="InvertImageNode", + display_name="Invert Image", + category="examples", + inputs=[ + io.Image.Input("image_in"), + ], + outputs=[ + io.Image.Output(display_name="image_out"), + ], + ) + + @classmethod + def execute(cls, image_in) -> io.NodeOutput: + image_out = 1 - image_in + return io.NodeOutput(image_out) + +class ExampleExtension(ComfyExtension): + async def get_node_list(self) -> list[type[io.ComfyNode]]: + return [ + InvertImageNode, + ] + +async def comfy_entrypoint() -> ExampleExtension: # ComfyUI calls this to load your extension and its nodes. + return ExampleExtension() +``` + +### Main properties + +Every custom node is a Python class inheriting from `io.ComfyNode`, with the following key properties: + +#### define_schema + +`define_schema`, as the name suggests, defines the schema for the node. The class method returns an `io.Schema` object which contains the node's metadata, inputs and outputs. + +The schema fields are: + +- `node_id`: a unique identifier for the node, used in the API and workflow JSON. +- `display_name`: the friendly name shown in the UI. Optional; defaults to `node_id`. +- `category`: where the node is found in the ComfyUI **Add Node** menu. Submenus can be specified as a path, eg. `examples/trivial`. +- `inputs`: a list of input objects. +- `outputs`: a list of output objects. +- `description`: the tooltip shown when hovering over the node. Optional. +- `search_aliases`: a list of alternative names users might search for when looking for this node. Optional. + +Each input is an object like `io.Int.Input("count", default=1, min=0, max=4096)`. The available input types are `io.Image`, `io.Mask`, `io.Model`, `io.VAE`, `io.CLIP`, `io.Conditioning`, `io.Latent`, `io.Audio`, `io.Int`, `io.Float`, `io.String`, `io.Combo`, `io.Boolean`, and more. See the [Datatypes](/custom-nodes/v3/backend/datatypes) page for details. + +As in V1, `define_schema` is a `@classmethod` so that widget options (like the name of the checkpoint to be loaded) can be computed at runtime. Let's go into this more later. + +Outputs are listed as `io.Image.Output()` and so on, with an optional `display_name` to label the output in the UI. + +#### execute + +The execution function is fixed to the name `execute`, and is a class method. It is called with named arguments matching the input ids defined in the schema. + +The function returns an `io.NodeOutput`, wrapping the result values. If the node has multiple outputs, pass them as multiple arguments: + +```python +return io.NodeOutput(model, clip, vae) +``` + +If the node has no outputs, return `io.NodeOutput()` (or simply nothing after processing side effects, but the return value must be `io.NodeOutput`, so `return` a bare `io.NodeOutput()`). + +### Execution Control Extras + +A great feature of Comfy is that it caches outputs, +and only executes nodes that might produce a different result than the previous run. +This can greatly speed up lots of workflows. + +In essence this works by identifying which nodes produce an output (these, notably the Image Preview and Save Image nodes, are always executed), and then working +backwards to identify which nodes provide data that might have changed since the last run. + +Two optional features of a custom node assist in this process. + +#### is_output_node + +By default, a node is not considered an output. Set `is_output_node=True` in the schema to specify that it is. + +```python +return io.Schema( + node_id="SaveImage", + ... + is_output_node=True, +) +``` + +#### fingerprint_inputs + +By default, Comfy considers that a node has changed if any of its inputs or widgets have changed. +This is normally correct, but you may need to override this if, for instance, the node uses a random +number (and does not specify a seed - it's best practice to have a seed input in this case so that +the user can control reproducibility and avoid unnecessary execution), or loads an input that may have +changed externally, or sometimes ignores inputs (so doesn't need to execute just because those inputs changed). + +`fingerprint_inputs` (formerly `IS_CHANGED` in V1) receives the same arguments as `execute` and returns a +value that Comfy compares with the one returned in the previous run. If the value differs, the node is executed. + +The name of this method was misleading in V1: `IS_CHANGED` is not "changed"; it is a cache key. Returning `True` every time makes the node run only once. + +A good example of actually checking for changes is the code from the built-in LoadImage node, which loads the image and returns a hash: + +```python + @classmethod + def fingerprint_inputs(s, image): + image_path = folder_paths.get_annotated_filepath(image) + m = hashlib.sha256() + with open(image_path, 'rb') as f: + m.update(f.read()) + return m.digest().hex() +``` + +To specify that your node should always be considered to have changed (which you should avoid if possible, since it +stops Comfy optimising what gets run), return a value that is never equal to the previous one, such as `float("NaN")`. + +#### not_idempotent + +If your node produces an output that depends on something other than its inputs (for example, a random number without a seed), set `not_idempotent=True` in the schema. This tells Comfy to skip the fast-path that assumes identical inputs produce identical outputs. + +### Other schema flags + +There are several other flags that can be used to modify how Comfy treats a node: + +- `is_deprecated`: flags the node as deprecated, telling users to find alternatives. +- `is_experimental`: flags the node as experimental, warning users that it may change. +- `is_input_list`: controls sequential processing of data, described [later](./lists). +- `hidden`: a list of hidden inputs (see [Hidden Inputs](./more_on_inputs#hidden-inputs)). +- `enable_expand`: allows the node to expand into a subgraph (see [Node Expansion](/custom-nodes/backend/expansion)). +- `accept_all_inputs`: passes all inputs that are not defined in the schema through to `execute` (see [Dynamically created inputs](./more_on_inputs#dynamically-created-inputs)). + +### validate_inputs + +If a class method `validate_inputs` is defined, it will be called before the workflow begins execution. +`validate_inputs` should return `True` if the inputs are valid, or a message (as a `str`) describing the error (which will prevent execution). + +#### Validating Constants + +Note that `validate_inputs` will only receive inputs that are defined as constants within the workflow. Any inputs that are received from other nodes will *not* be available in `validate_inputs`. + +`validate_inputs` is called with only the inputs that its signature requests (those returned by `inspect.getfullargspec(obj_class.validate_inputs).args`). Any inputs which are received in this way will *not* run through the default validation rules. + +For example, in the following snippet, the front-end will use the specified `min` and `max` values of the `foo` input, but the back-end will not enforce it. + +```python +from comfy_api.latest import io + +class CustomNode(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="CustomNode", + inputs=[ + io.Int.Input("foo", min=0, max=10), + ], + outputs=[], + ) + + @classmethod + def validate_inputs(cls, foo): + # YOLO, anything goes! + return True +``` + +Additionally, if the function takes a `**kwargs` input, it will receive *all* available inputs and all of them will skip validation as if specified explicitly. + +#### Validating Types + +If the `validate_inputs` method receives an argument named `input_types`, it will be passed a dictionary in which the key is the name of each input which is connected to an output from another node and the value is the type of that output. + +When this argument is present, all default validation of input types is skipped. Here's an example making use of the fact that the front-end allows for the specification of multiple types: + +```python +from comfy_api.latest import io + +class AddNumbers(io.ComfyNode): + @classmethod + def define_schema(cls) -> io.Schema: + return io.Schema( + node_id="AddNumbers", + inputs=[ + io.MultiType.Input( + io.Int.Input("input1", min=0, max=1000), + types=[io.Int, io.Float], + ), + io.MultiType.Input( + io.Int.Input("input2", min=0, max=1000), + types=[io.Int, io.Float], + ), + ], + outputs=[io.Int.Output()], + ) + + @classmethod + def validate_inputs(cls, input_types): + # The min and max of input1 and input2 are still validated because + # we didn't take `input1` or `input2` as arguments + if input_types["input1"] not in (io.Int.io_type, io.Float.io_type): + return "input1 must be an INT or FLOAT type" + if input_types["input2"] not in (io.Int.io_type, io.Float.io_type): + return "input2 must be an INT or FLOAT type" + return True +``` diff --git a/docs.json b/docs.json index b6a227405..947363a1e 100644 --- a/docs.json +++ b/docs.json @@ -3424,7 +3424,7 @@ "custom-nodes/overview", "custom-nodes/walkthrough", { - "group": "Python (Backend)", + "group": "Python (Backend) · V1", "pages": [ "custom-nodes/backend/server_overview", "custom-nodes/backend/lifecycle", @@ -3439,6 +3439,17 @@ "development/comfyui-server/execution_model_inversion_guide" ] }, + { + "group": "Python (Backend) · V3", + "pages": [ + "custom-nodes/v3/backend/server_overview", + "custom-nodes/v3/backend/datatypes", + "custom-nodes/v3/backend/more_on_inputs", + "custom-nodes/v3/backend/lazy_evaluation", + "custom-nodes/v3/backend/lifecycle", + "custom-nodes/v3/backend/lists" + ] + }, { "group": "JavaScript (UI)", "pages": [ @@ -6656,7 +6667,7 @@ "zh/custom-nodes/overview", "zh/custom-nodes/walkthrough", { - "group": "Python（后端）", + "group": "Python（后端）· V1", "pages": [ "zh/custom-nodes/backend/server_overview", "zh/custom-nodes/backend/lifecycle", @@ -9961,7 +9972,7 @@ "ja/custom-nodes/overview", "ja/custom-nodes/walkthrough", { - "group": "Python（バックエンド）", + "group": "Python（バックエンド）· V1", "pages": [ "ja/custom-nodes/backend/server_overview", "ja/custom-nodes/backend/lifecycle", @@ -13179,7 +13190,7 @@ "ko/custom-nodes/overview", "ko/custom-nodes/walkthrough", { - "group": "Python (백엔드)", + "group": "Python (백엔드) · V1", "pages": [ "ko/custom-nodes/backend/server_overview", "ko/custom-nodes/backend/lifecycle", @@ -14364,4 +14375,4 @@ "destination": "/tutorials/partner-nodes/wan/wan3-0" } ] -} +} \ No newline at end of file