Hydra 3.5 is a finetune of the SigLIP2 image classifier with a custom classifier head, designed to predict 8,886 popular tags from e621.
It is the successor to JTP-3 Hydra.
For Windows, simply download install.bat in the folder where you want Hydra 3.5 to be installed, and double-click it to run the full installer.
You may need to left-click it and select "Run as Administrator..." if Python isn't already installed.
Alternatively, you can clone the repository first and install manually in the same way as on Linux.
Once installed, you can run the graphical autotagger by double-clicking gui.bat.
Linux Installation
If you have Git+LFS installed, download the repository using git clone --depth=1 https://huggingface.co/RedRocket/Hydra.
If your OS Python install is not 3.11 or above, install a more recent version of Python according to your distribution's instructions and use that python to create the venv.
You can check your version of python with python -V.
When you launch the GUI, the model begins loading automatically. Avoid running multiple GUIs at the same time, as each copy will load the entire model.
The UI supports drag-and-drop of files, folders, and web links where appropriate.
Hover your mouse over most UI elements to show an explanatory tool tip and view the assigned keyboard shortcut, if any.
Right-click tags in the Model Info and Stats tabs to view additional actions for that tag.
Bulk Autotagging Tab
You can drag and drop a folder or a file inside it in order to select that folder.
Folder — Enter a path to the directory containing your images, or click … to browse for one.
Subfolders — Check to process images in subfolders of the selected folder too.
Keep — Controls which labels in any existing caption files are retained:
none — Completely replace existing captions.
unknown — Keep existing labels not supported by the model.
all — Keep all existing labels, even if the model disagrees.
skip — Skip images which already have a caption file.
Classify — Click to run the autotagger. The progress bar fills as each image is completed and the Stats tab will show live stats. Click Stop to cancel at any time.
Log — Displays status messages, errors, and a summary after each run. The summary shows how many files were processed, how many failed, total elapsed time, and speed.
Bulk Autotagging Stats
After each run, shows a tag list with the number of times each tag was applied to an image.
The list does not count tags only included because they were in the Prefix.
Right-clicking a tag opens a context menu with additional actions for that tag, such as adding it to the list of excluded tags before another run.
Single Image Tab
You can drag and drop a file or link in order to select that image. If you drag and drop a folder, the first image in that folder will be selected.
Open — Browse for an image or folder to open.
← (Previous) — [Alt+LeftArrow] Move to previous image in folder. Not visible if viewing a web link. → (Next) — [Alt+RightArrow] Move to next image in folder.
Rerun — [F2] Re-run the classifier, replacing the current list of tags. If Shift is held, the current list of tags will be preserved instead.
Visualize — [F3] Select a tag for CAM attention visualization. If Shift is held, a Principal Component Analysis will be run instead.
Clean — [Alt+L] Re-formats the list of tags to remove empty or duplicate tags. If Shift is held, the list will also be sorted.
Single Image Stats
Presents an alternative editor for the list of tags, with predicted probabilities visible.
Manually added tags not predicted by the model are shown at the top of the list.
Tags not recognized by the model will be shown in red, along with up to three suggestions of tags which are recognized by the model.
Right-clicking a tag will show a context menu with additional options for the tag and their keyboard shortcuts. For example Alt-Clicking a tag will run the Attention Visualization on that tag.
If Ctrl is held while checking or unchecking a tag the action will be implication-aware.
Add Implied — [Alt+A] Add all missing tags that are implied by the currently selected tags.
Copy CSV — [Alt+C] Copy the list of tags as a CSV. If Shift is held, save as a file instead.
The bottom bar shows the accuracy of the model given the list of tags. That is to say, the number of checked predicted tags out of the total number of supported tags in the list.
Classification Tab
Controls how tags are selected from the model's raw output. These settings apply to both Bulk Autotagging and Single Image mode.
Calibration Metric — Configures the model calibration, which automatically sets the thresholds for each individual tag for the best performance:
Confidence — Presents a familiar 0.0 (most speculative tags) to 1.0 (most confident tags) slider.
Under the hood, this is translated logarithmically to an F Score between 4.0 and 0.25.
F Score — Calibrates the model based on precision (confident tags) and recall (speculative tags).
The number beside this is the β value; 1.0 strikes an even balance and values below 1.0 favor precision (fewer tags).
A β of 0.25 (¼) is the opposite of a beta of 4.0.
CSI — Uses the Critical Success Index, an alternative metric similar to F Score.
Threshold — A hard lower bound on tag precision. Increase this to remove rarer, less-accurate tags.
Implications — Controls how tag implications are applied to the raw model outputs.
For the most intuitive results, select inherit to be permissive or enforce-inherit to be strict.
To still see the raw model outputs, use preserve or enforce instead.
For a more cautious assessment, swap inherit for constrain.
preserve — Tags are preserved if they are implied by another tag.
inherit — Tags are preserved and inherit the highest probability among the tags that imply them.
constrain — Tags are preserved and inherit the lowest probability among the tags they imply.
enforce — Tags are removed unless all the tags they imply are present.
remove — Exclude all implied tags.
constrain-remove — Combination of constrain followed by remove.
enforce-inherit — Combination of enforce followed by inherit.
enforce-constrain — Combination of enforce followed by constrain.
enforce-remove — Combination of enforce followed by remove.
off — Raw model output with no implications applied. Probabilities and tags may be inconsistent.
Exclude Categories — Check a category's box to remove all tags belonging to it from the output.
Exclude Tags — A free-text box where you can list specific tags to suppress, in e621 format, space- or line-separated.
Use Save/Load to save to or load from a .txt file.
Exclusive Groups — A free-text box for defining mutually exclusive tag groups, one group per line.
When tags in the same group are predicted, only the highest-scoring one is kept.
This box is pre-filled with a reasonable default based on the model, but it won't be suitable for absolutely all cases.
Use Save/Load to save to or load from a .txt file.
Format Tab
Controls how output tags are written to caption files and the format of the tag list in Single Image mode.
Prefix — A string prepended to every output tag string (e.g. a style trigger word followed by a comma).
If the prefix contains tags, those tags will not be repeated.
Spaces — Replaces underscores with spaces in tag names (example_tag → example tag).
"by" Prefix — Prepends by_ to artist tags (artist_name → by_artist_name).
Only available when the loaded model includes an artist category, which Hydra 3.5 doesn't support by default.
Prompt Syntax — Escapes parentheses and other prompt syntax for direct use with diffusion models ((suffix) → \(suffix\)).
Generally this should be off for LoRA training.
Shuffle Tags — Randomises the order of output tags on each run. The Prefix is never shuffled.
Aliases — A free-text box for tag renaming rules, one e621-format alias per line in the order MODEL_TAG REPLACEMENT_TAG.
Use Save/Load to save to or load from a .txt file.
The included data/aliases-2024-04-07.txt file contains 120+ automatically-loaded aliases that improve compatibility with diffusion models such as NoobAI.
Settings Tab
Controls performance and resource usage.
Batch Size — Number of images processed in each GPU batch in Bulk Autotagging mode.
Higher values generally improve speed but use more VRAM.
Loader Workers — Number of processes used to load and preprocess images in Bulk Autotagging mode.
If set to 0, the UI is more likely to freeze while classifying.
-1 creates one loader per cpu core, up to 16 or one per 4 images.
-2 creates one loader per cpu core.
The number of workers is automatically capped to half the number of images.
NaFlex Seqlen — Controls the maximum number of 16x16 pixel image patches that the model sees.
It's best not to change this, but 256 should work okay if you are very compute constrained.
Varlen Attention — Enables variable-length attention for more efficient processing.
Only available if the installed version of pytorch supports it.
Shared Memory — Uses shared memory for the data loader.
Disable this if shared memory is somehow causing problems.
Model Info Tab
Displays the loaded model's name and a full list of all tags it knows, each annotated with its category and the tags it implies and is implied by and its calibrated threshold.
Right-click a category or tag to see a menu of additional actions.
Using inference.py (or inference.bat)
$ python inference.py --help
usage: inference.py [-b BATCH_SIZE] [-w N_WORKERS] [-d DEVICE] [-o CSV] [-r] [...] [--] PATH...
input:
PATH Paths to files and directories to classify.
-r, --recursive Classify directories recursively. Dotfiles will be ignored.
-k, --keep [none|unknown|all|skip]
Labels to retain from any existing caption files. If not specified, defaults to 'none'. If no argument is provided, defaults to 'unknown'.
output:
-h, --help Show this help message and exit.
-o, --output PATH Path for CSV output, or '-' for standard output. If not specified, individual .txt caption files are written.
-p, --prefix PREFIX Prefix all .txt caption files with the specified text. If the prefix matches a tag, the tag will not be repeated.
-s, --shuffle Shuffle tag strings.
-u, --underscores Do not convert underscores to spaces.
-P, --prompt Escape prompt syntax such as parenthesis.
-a, --alias OLD NEW Change the name of a tag.
-A, --aliases PATH Path to tag alias file, with one space-separated alias per line. May be specified multiple times.
-B, --category-prefix CATEGORY PREFIX
Define a prefix appended to all tags with the specified category. May be specified multiple times.
classification:
-m, --metric METRIC Calibration metric. (Default: f1.0@0.1)
-i, --implications MODE
Automatically apply implications. (Default: inherit)
-x, --exclude-tag TAG
Exclude the specified tag. May be specified multiple times.
-X, --exclude-tags PATH
Load a list of tags to exclude from the specified file. May be specified multiple times.
-C, --exclude-category CATEGORY
Exclude the specified category of tags. May be specified multiple times.
-g, --exclusive-group [[[!]TAG ...] [!]TAG:] TAG TAG [TAG ...]
Define a group of mutually-exclusive tags with an optional precondition. May be specified multiple times.
-G, --exclusive-groups PATH
Load a list of mutually-exclusive groups from the specified file, one per line. May be specified multiple times.
model:
-M, --model PATH Path to model file. (Default: models/hydra-3.5.safetensors)
-D, --metadata PATH Metadata directory for legacy JTP-3 models. (Default: ./data)
-e, --extension PATH Path to extension. May be specified multiple times. If a directory is specified, all extensions in the specified directory are loaded. (Default: extensions/<model_name>)
-E, --no-default-extensions
Do not load extensions by default.
execution:
-b, --batch-size BATCH_SIZE
Batch size.
-w, --workers N_WORKERS
Number of dataloader workers, capped to half the number of images. 0 loads all images serially in-process. -1 creates one loader per cpu core, up to 16 or one per 4 images. -2 creates one loader per cpu core.
--no-shm Disable shared memory between workers.
-V, --varlen Use optimized varlen attention. (Requires flash attention support.)
-S, --seqlen SEQLEN NaFlex sequence length. (Default: 1024)
-d, --device DEVICE Torch device. (Default: cuda)
-c, --compile Compile the model for maximum performance.
METRIC: metric[arg][@min_prec]
metric: f, csi
arg: Optional positive number trading off precision and recall, default 1.0.
min_prec: Optional minimum precision floor, default 0.0.
MODE:
preserve Tags are preserved if they are implied by another tag.
inherit Tags are preserved and inherit the highest probability among the tags that imply them.
constrain Tags are preserved and inherit the lowest probability among the tags they imply.
enforce Tags are removed unless all the tags they imply are present.
remove Exclude all implied tags.
constrain-remove Combination of constrain followed by remove.
enforce-inherit Combination of enforce followed by inherit.
enforce-constrain Combination of enforce followed by constrain.
enforce-remove Combination of enforce followed by remove.
off Raw model output with no implications applied.
CATEGORY:
general artist contributor copyright character species invalid meta lore color rating
(or other categories supported by the model or extensions)
Avoid running multiple copies of inference.py at once, as each copy will load the entire model.
Using the HTTP Service. (service.py/service.bat)
By default, the FastAPI service runs on port 8080 and is only accessible locally.
Explorable API documentation is available at /docs, including a live demo.
If you are writing a client for the service, strongly consider resizing large images before sending them over the network.
You should use the same resizing technique as hydra/image.py for best performance.
At the very least, ensure the alpha channel is either preserved or composited with white.
$ python service.py --help
usage: service.py [-i IP] [-p PORT] [-b BATCH_SIZE] [-d DEVICE] [...]
service:
-h, --help Show this help message and exit.
-i, --ip IP Service IP address. (Default: 127.0.0.1)
-p, --port PORT Service port. (Default: 8080)
model:
-M, --model PATH Path to model file. (Default: models/hydra-3.5.safetensors)
-D, --metadata PATH Metadata directory for legacy JTP-3 models. (Default: ./data)
-e, --extension PATH Path to extension. May be specified multiple times. If a directory is specified, all extensions in the specified directory are loaded. (Default: extensions/<model_name>)
-E, --no-default-extensions
Do not load extensions by default.
execution:
-b, --batch-size BATCH_SIZE
Maximum batch size. (Default: 1)
-V, --varlen Use optimized varlen attention. (Requires flash attention support.)
-S, --seqlen SEQLEN NaFlex sequence length. (Default: 1024)
-d, --device DEVICE Torch device. (Default: cuda)
-c, --compile Compile the model for maximum performance.
GET /info POST /classify?calibration=METRIC&implications=MODE image/*
The calibration METRIC and implications MODE are optional, and have the same meaning and defaults as in inference.py.
Other Utilities
Use extract_tags.py to unpack the tag metadata embedded in the model file for easier use with other software.
You can also use the Python API provided by the hydra subpackage to work with tags directly, and you can use load_model(metadata_only=True) if inference is not necessary.
Use check_tags.py to help with building new compatibility alias files. You may need to adapt the code somewhat.
Usage Notes
The model predicts 8,884 e621 tags, as well as the added rating meta-tags safe, questionable, and explicit.
The model is trained with implications, but its raw predictions are not constrained.
The model is trained on images on e621 only, and not on photographs of people or real animals.
While it has retained some ability to classify photos, this is not in any way supported.
The interactive interfaces perform automatic calibration of the model using embedded validation data.
The model sees all transparency as a white background.
By default, the model "sees" at a resolution equivalent to 512x512 pixels (0.26MP).
Because the model uses NAFlex, image aspect ratios are preserved, meaning that they are not stretched or distorted when processing.
Extensions and Training
Support for training extensions will be added at a later date if there is community interest and when time permits.
Technical Notes
The model consists of SigLIP2 So400m Patch16 NAFlex followed by a custom cross-attention transformer with learned per-tag queries, SpGLU feedforward, and per-tag output heads.
The per-tag cross attention mechanism is the origin of the name "hydra".
Subject to the preprocessing mentioned below, the initial set of training tags was all general tags with at least 1,200 examples, all species and character tags with at least 500 examples, a semi-automated selection of copyright and meta tags, and a handful of manually-selected lore tags which are sometimes discernible from the image.
Extensive semi-manual dataset curation was used to improve the quality of the training data.
The dataset preprocessing code consists of over 12,000 lines of code and data files.
In addition to correcting implications, manually-defined rules are used to detect common scenarios of missing, incomplete, or contradictory tagging and to selectively mask individual tags on a per-dataset-item basis.
This is responsible for Hydra 3.5's excellent performance in detecting colors and "combo tags" such as male_feral.
Self-distillation cross entropy loss based on ASL was used to mitigate the effects of inconsistent labeling on e621 and the extreme class imbalance.
The dataset was sampled in mini-epochs according to a self-entropy metric.
Credits
RedHotTensors — All changes since JTP-3.
Renfald & Annonymous — Tag compatibility and testing.
Comet — Additional testing.
Google Gemini — Hero image.
