[Bug]: Different results with same workflow when using SDXL Attention Injection

[fappaz]

What happened?

I ran the same workflow in ComfyUI desktop and portable, with the same version of ComfyUI-layerdiffuse (latest version from the main branch, which I suppose is nightly 1.0.4) and got different results when using SDXL Attention Injection.

Log for the desktop version:
comfyui.log

I noticed the following warnings in the console for the desktop version, which doesn't happen in the portable version:

(...)
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_k.weight
loaded completely 5388.290207672119 4897.0483474731445 True
  0%|          | 0/4 [00:00<?, ?it/s]C:\Users\SimplesmenteIA\Documents\ComfyUI\.venv\Lib\site-packages\torchsde\_brownian\brownian_interval.py:608: UserWarning: Should have tb<=t1 but got tb=14.614644050598145 and t1=14.614643.
  warnings.warn(f"Should have {tb_name}<=t1 but got {tb_name}={tb} and t1={self._end}.")
100%|██████████| 4/4 [01:09<00:00, 17.47s/it]
Requested to load AutoencoderKL
loaded completely 2048.9999771118164 159.55708122253418 True
100%|██████████| 8/8 [00:01<00:00,  5.74it/s]
Prompt executed in 108.38 seconds

When I use SDXL Conv Injection instead in the Layer Diffuse Apply node, there is some transparency in the result in both ComfyUI versions.

Steps to reproduce the problem

1. Download the first image attached above, then import it into ComfyUI portable version.
2. Run the workflow.
3. Repeat step 1, but import it into ComfyUI desktop version instead.
4. Run the workflow and notice how the image is different.
5. Change the config in the Layer Diffuse Apply node to SDXL Conv Attention, then run the workflow again to get a result with transparency.

What should have happened?

The same workflow should result in the same image in both ComfyUI versions.

Commit where the problem happens

ComfyUI portable: a3100c8452862e914996648e0fbc56098ab26b60
ComfyUI desktop: v0.3.29
ComfyUI-layerdiffuse: b4f6a9e024064a4489f774a8b91049ce0b606ea3 for both versions.

Sysinfo

OS: nt (Windows 10)
Graphics Card: NVIDIA GeForce RTX 3080 Laptop GPU

Console logs

See log file attached in the description above.

Workflow json file

ComfyUI Portable:

ComfyUI desktop:

Using SDXL Conv Injection instead:

Additional information

All workflows were tested with the same Layer Diffuse models. I also compared the ComfyUI-layerdiffuse folders for both ComfyUI with git diff, but they're identical, which leads me to believe this might be an incompatibility issue with ComfyUI desktop.

[fappaz]

I did some more investigation today, and noticed a few things (workflows are embedded in the attached images).

Instead of the standard KSampler, I used a SamplerCustom (keeping same seed, sampler, scheduler, etc) and disabled add noise, and finally got some transparency in the image. However, the result is still totally different compared to the one with same parameters generated in ComfyUI portable. Any idea why??

Also, I noticed that the "failed" generation looks pretty much the same as if Layer Diffuse was off. I tried changing the weight to the highest value possible (3.00) to see if it would make any difference, but there was almost no change.

And if that wasn't enough, I updated my ComfyUI portable to the same version as my ComfyUI desktop - both are in v0.3.31 - but the issue remains.

Layer Diffuse ON:

Layer Diffuse OFF (bypassed):

[fappaz]

Feeling I'm getting closer, but need your help to reproduce the issue.

When I install and enable Fooocus_Nodes, the transparency works as expected! As soon as I disable it and restart ComfyUI, it no longer works.

Now, this behavior implies that:

1. The failed generation is actually how Layer Diffuse works by default!
2. Fooocus_Nodes seems to implement some sort of patcher that intercepts the model (or sampling) even when none of its nodes are in the workflow. I've seen this happening several times before [1] [2] [3].

Any more ideas?

[fappaz]

@huchenlei can you please confirm whether you get the same image when you run the workflow below?

Also, do you know why having Fooocus_nodes enabled in ComfyUI changes the results of your nodes?

[nabileisa]

I have the same problem , when using SDXL, Attention Injection

patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.0.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.4.1.transformer_blocks.1.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.0.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn2.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn2.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.5.1.transformer_blocks.1.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.0.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.0.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.0.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.0.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.0.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.0.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.1.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.1.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.1.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.1.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.1.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.1.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.2.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.2.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.2.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.2.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.2.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.2.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.3.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.3.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.3.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.3.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.3.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.3.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.4.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.4.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.4.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.5.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.5.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.5.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.5.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.5.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.5.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.6.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.6.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.6.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.6.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.6.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.6.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.7.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.7.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.7.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.7.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.7.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.7.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.8.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.8.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.8.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.8.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.8.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.8.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.9.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.9.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.9.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.9.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.9.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.7.1.transformer_blocks.9.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.0.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.0.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.0.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.0.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.0.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.0.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.1.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.1.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.1.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.1.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.1.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.1.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.2.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.2.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.2.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.2.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.2.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.2.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.3.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.3.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.3.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.3.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.3.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.3.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.4.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.4.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.4.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.4.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.4.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.4.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.5.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.5.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.5.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.5.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.5.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.5.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.6.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.6.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.6.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.6.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.6.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.6.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.7.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.7.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.7.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.7.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.7.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.7.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.8.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.8.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.8.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.8.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.8.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.8.attn2.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.9.attn1.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.9.attn1.to_k.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.9.attn1.to_v.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.9.attn1.to_out.0.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.9.attn2.to_q.weight
patch type not recognized lora diffusion_model.input_blocks.8.1.transformer_blocks.9.attn2.to_out.0.weight

in the console repeating with each step and the output image is not transparent , also using the latest portable version of comfy and comfyui-layerdiffuse
sd1.5 works fine although i suffer from this bug as well #104 but the fix in the comments seem to work .

[fappaz]

I'm getting closer but still need help from @huchenlei

I "fixed" the issue by patching ComfyUI, but I believe this is not the right way. Instead, I think this layerdiffuse project is setting the patch info incorrectly (or used to be correct before ComfyUI changed the way they calculate weights), making ComfyUI unable to load the LoRA properly (hence the error messages).

Here's how I did it: In ComfyUI's comfy\lora.py, I edited the calculate_weight function and added the following condition in line 382:

# Adapted from `custom_nodes\fooocus_nodes\py\modules\patch.py`
elif patch_type == "lora":
            mat1 = comfy.model_management.cast_to_device(v[0], weight.device, intermediate_dtype)
            mat2 = comfy.model_management.cast_to_device(v[1], weight.device, intermediate_dtype)
            alpha = 1.0
                
            try:
                lora_diff = torch.mm(mat1.flatten(start_dim=1), mat2.flatten(start_dim=1)).reshape(weight.shape)
                weight += function(((strength * alpha) * lora_diff).type(weight.dtype))
            except Exception as e:
                print("ERROR {} {} {}".format(patch_type, key, e))

Now the error messages are gone and the output is correct. However, as I said, I believe this must be fixed in this project instead of ComfyUI's. What do you think?

[Sisimshow]

I just want to comment that I'm having the same issue. I am using Comfy through SwarmUI, but I used to be able to use Attention Injection and now I can't because the transparency layer doesn't properly generate. Here's another example of it not properly working with one of the example workflows.

[Geeknasty]

I think the format of layer_xl_transparent_attn.safetensors is not compatible after PR #7540

I converted the layer_xl_transparent_attn.safetensors and uploaded it here: https://huggingface.co/Geeknasty/LayerDiffusion-SDXL-Attn-Diff
It is now the same format as the other files in original repo: https://huggingface.co/LayerDiffusion/layerdiffusion-v1

I tested it with ComfyUI v0.3.45 and latest comfyui-layerdiffuse version = "1.0.4"

[fappaz]

I can confirm this works, thanks heaps!

@huchenlei would you be so kind to update the project so the custom node can auto download the model with the fix?

I think the format of layer_xl_transparent_attn.safetensors is not compatible after PR #7540

I converted the layer_xl_transparent_attn.safetensors and uploaded it here: https://huggingface.co/Geeknasty/LayerDiffusion-SDXL-Attn-Diff It is now the same format as the other files in original repo: https://huggingface.co/LayerDiffusion/layerdiffusion-v1

I tested it with ComfyUI v0.3.45 and latest comfyui-layerdiffuse version = "1.0.4"