TinyBreaker (prototype1)

 

 

Overview

Key Features

• Hybrid Two-Step Architecture: Combines PixArt-Sigma as the base model with a refiner based on Photon (or any SD1.x model), both chosen for their low GPU consumption.
• Efficient Parameter Usage: The base model’s 0.6 billion parameters enable high-quality image generation with minimal computational overhead.
• Fast Performance: Produces high-quality 1536×1024 images in ~15 seconds on an NVIDIA RTX 3080 GPU, with ongoing work to cut generation times to under 10 seconds.
• High Prompt Adherence: Generates images that closely match user prompts and expectations, thanks to the robust performance of the PixArt-Sigma model and the T5 text encoder.
• Optimized Latent Space Processing: Leverages Tiny Autoencoders for efficient latent space conversion.

Usage Requirements

Limitations

• Text Generation: Generating legible text within images is a challenge due to PixArt's training limitations. Enhancements in this area may require extensive retraining.
• Human Anatomy in Complex Poses: While the model performs reliably with standard poses (e.g., standing, facing the camera), it struggles with anatomical accuracy in poses that require more complex or dynamic actions.
• Complex Human Interactions: The model has difficulty generating detailed scenes involving intricate interactions among people, as well as interactions between people and objects, such as collaborative tasks or dynamic object manipulation.

Future Directions

Acknowledgments

• I extend my sincere thanks to the PixArt-Σ developers for their exceptional model, which has been vital to this project's development.
  PixArt-Σ GitHub Repository | PixArt-Σ Hugging Face Model | PixArt-Σ arXiv Report
• Additional thanks to Ollin Boer Bohan for the Tiny AutoEncoder models, which offer efficient latent image processing and served as the foundation for the encoding, decoding, and transcoding operations in TinyBreaker.
  Tiny AutoEncoder GitHub Repository

Resources

• TinyBreaker on CivitAI: A hub for exploring generated images, prompts, and workflows created by me and the community, showcasing the model's output quality.
• ComfyUI-TinyBreaker: Nodes and workflows for ComfyUI to experiment with the model's capabilities.
• TinyBreakerTools: Tools I'm building for the model, mainly to create the safetensors file for TinyBreaker.
• AbominableWorkflows: A predecessor of TinyBreaker. My first experiment combining PixArt-Sigma and Photon without Python code, using only standard nodes from ComfyUI.