This project aims to train an image generation model to learn and reproduce a new character concept that is not present in the original base model. The objective is to extend the model’s visual capabilities while maintaining stylistic consistency and coherence in the generated outputs. Special thanks to Ostris for providing the AI Toolkit used throughout the training process. Full credit for the Flux model architecture goes to Black Forest Labs.

All content produced using this model is intended solely for non-commercial use.

The character to is going to be introduced is Vinni Pukh, the Soviet adaptation of Winnie-the-Pooh. This iconic character originates from a series of animated short films produced between 1969 and 1972 by Fyodor Khitruk at the Soyuzmultfilm studio. Unlike Disney’s Western version, Vinni Pukh is characterized by a minimalist visual style, absurd humor, and a more subtle philosophy—hallmarks of Soviet animation of that era. Vinni is portrayed as naive, clumsy, yet deeply endearing. The films, animated with expressive motion and featuring gouache-painted backgrounds, explore themes of friendship, curiosity, and everyday whimsy with gentle irony.

To avoid any interference between what the model already knows about Vinni Pukh and the new concept aimed to be introduced, the character was rename Biggyboo.

The trained model is expected to:

The complete training data is available in the repository. The dataset consists of low-resolution images collected from two sources: public Google Image search results and manual screenshots extracted from the animated movies. In total, 11 images were gathered to serve as the basis for style and character learning during training.

AI Toolkit works best with image captioning when training a model on a concept. It is recommended to annotate the images using only the character’s name, without specifying invariant attributes. This allows the model to learn the character’s identity without restricting it to a single situation or appearance. The captioning model Florence-2 was used for this purpose, with the highest level of detail. After the captions were generated, they were manually reviewed and adjusted when necessary. All references to the original character traits of Vinni Pukh were removed, and the concept name Biggyboo was appended to each caption.

The LoRA training using AI Toolkit was carried out on Runpod, a cloud computing platform designed for AI, machine learning, and other compute-intensive workloads. Runpod provides scalable, high-performance GPU and CPU resources with pay-as-you-go pricing, making it well-suited for running large models, training neural networks, or deploying applications efficiently and flexibly.

For this project, we used an NVIDIA RTX 6000 Ada GPU, featuring 48 GB of VRAM and 62 GB of system RAM. The cost of this GPU instance was $0.77 per hour.

The selected Runpod template was “ComfyUI with Flux.1-dev”, which automatically installs:

A Hugging Face token must be provided as an environment variable to allow model downloads. Additionally, the container’s storage volume was expanded to 150 GB to accommodate the dataset, checkpoints, and training outputs.

The results of the fine-tuned model show that it successfully learned the intended concept. Simply mentioning the character’s name Biggyboo in a prompt consistently triggers its appearance in the generated image. However, in addition to learning the character, the model also absorbed elements of the Soviet animation style present in the training data.

Performance is also noticeably better when the actions, environments, and expressions described in the prompt remain close to those encountered during training. It may be worth experimenting with a dataset in which the character is completely isolated from its environment, or where the character is represented in a variety of styles, in order to reduce unwanted stylistic influence on the model.

This proof of concept provided valuable insights into fine-tuning an image generation model. One of the key lessons learned is that, although Flux is relatively easy to train, it is essential to clearly define your objectives before starting. Without a well-defined goal, the model may either overfit or fail to accurately capture the intended concept.

You should use Biggyboo to trigger the image generation.

Weights for this model are available in Safetensors format.

Download them in the Files & versions tab.

For more details, including weighting, merging and fusing LoRAs, check the documentation on loading LoRAs in diffusers