You need to first download the FaceCaption-15M from our huggingface and then apply for access to the original Laion-face dataset by completing the required agreement (github). Once approved, refer to the information available on HuggingFace to obtain the corresponding image-text pairs.
Fig.1:(a). Same color represents shared parameters. “12x” stands for 12-layer transformer modules. (b), (c) and (d) FLIP-based model are applied to the tasks of text-image retrieval, facial attributes prediction and sketch less facial image retrieval, respectively.
Training
Coming soon......（Only for the datasets been published, the code of training is meaningful.）
python pretrain.py > log.log
Pre-trained Models
We provide pretrained model weights :
FLIP Base —— click here
FLIP Large —— coming soon......
Table 1: Comparison with other classical pretrained models. All pretrained model backbones are frozen, with only the linear layer being fine-tuned. † represents the model pretrained on the LAION-Face [86] dataset; * represents the model pretrained on the FaceCaption dataset constructed without using LLM text generation.
Task2: Facial Attributes Prediction
Table 2: Comparison with other classical models. † represents the model pre-trained on the original LAION-Face dataset.
image-20240318101126897
Task3: Sketch Less Facial Image Retrieval
Table 3: Comparative results with different baseline methods. † represents the model pre-trained on the LAION-Face dataset.
image-20240318101633671
image/png
Fig.2:Demonstration of our FLIP-based model on the SLFIR task. Both methods can retrieve the target face photo from the top-5 list using a partial sketch. Our proposed FLIP-based model can achieve this using fewer strokes than the baseline. The number at the bottom denotes the rank of the paired (true match) photos at every stage.
1@misc{dai202415mmultimodalfacialimagetext,
2 title={15M Multimodal Facial Image-Text Dataset},
3 author={Dawei Dai and YuTang Li and YingGe Liu and Mingming Jia and Zhang YuanHui and Guoyin Wang},
4 year={2024},
5 eprint={2407.08515},
6 archivePrefix={arXiv},
7 primaryClass={cs.CV},
8 url={https://arxiv.org/abs/2407.08515},
9}1011@article{dai2026facecaption15m,
12 title={FaceCaption-15M: Benchmarking and Enhancing Facial Vision-Language Models},
13 author={Dawei Dai and YuTang Li and YingGe Liu and Mingming Jia and Zhang YuanHui and Guoyin Wang},
14 journal={IEEE Transactions on Circuits and Systems for Video Technology},
15 year={2026},
16 doi={10.1109/TCSVT.2026.3713342},
17 keywords={Modeling;Faces;Training;Conferences;Computers;Computer vision;Learning (artificial intelligence);Pattern recognition;Educational institutions;Machine learning;Image-Text Dataset;Face Task},
18}19
