Current version is still WIP. I'm not 100% finished with this current version. There's definitely things I need to fix / limitations with this version. But, I'm uploading it for now, just to get something out.
If using sage-attention, make sure to switch to sage-attention-triton (you can use KJ nodes to do this). As the default sage-attention doesn't support the cross-attention mask.
PDG custom node included at extension/block_drop_guidance.py, used to increase sharpness of outputs. (Just place in extensions folder of ComfyUI)
Current final finetune was trained with slight PDG dropout (0.05).
Adapter changes:
Cross-attention and adapter masking of padding tokens.
(Padding is now used to pad to nearest 77 chunk as sage-attention relies on this)
Removed the additional positional embeddings from the adapter. This was a mistake on my part to add to the original. The position of the text now has less of an impact on the output generation.
Works with, and trained with prompts up to 1024 tokens.
