🌟 This is a small (only 1.2B parameters) visual language model on Hugging Face that responds to Traditional Chinese instructions given an image input! 🌟
✨ Developed compatible with the Transformers library, TaiVisionLM is quick to load, fine-tune, and use for lightning-fast inferences without needing any external libraries! ⚡️
Ready to experience the Traditional Chinese visual language model? Let's go! 🖼️🤖
This model is a multimodal large language model that combines SigLIP as its vision encoder with Tinyllama as its language model. The vision projector connects the two modalities together.
Its architecture closely resembles PaliGemma.
We trained the vision projector and language model using LoRA using 1M image-text pairs to align visual and textual features.
This model is the finetuned version of benchang1110/TaiVisionLM-base-v1. We fintuned the model using 1M image-text pairs. The finetuned model will generate a longer and more detailed description of the image.
Task Specific Training
The aligned model undergoes further training for tasks such as short captioning, detailed captioning, and simple visual question answering.
We will undergo this stage after the dataset is ready!
In Transformers, you can load the model and do inference as follows:
IMPORTANT NOTE: TaiVisionLM model is not yet integrated natively into the Transformers library. So you need to set trust_remote_code=True when loading the model. It will download the configuration_taivisionlm.py, modeling_taivisionlm.py and processing_taivisionlm.py files from the repo. You can check out the content of these files under the Files and Versions tab and pin the specific versions if you have any concerns regarding malicious code.
