Llama 3.1 Vision by Capx AI

Directions to Run Inference:

• Clone our fork of the Bunny by BAAI repository here: https://github.com/adarshxs/Capx-Llama-3.1-Carrot
• Create a conda virtual environment
  bash

  1conda create -n capx python=3.10 2conda activate capx
• Install the following
  bash

  1pip install --upgrade pip # enable PEP 660 support 2pip install transformers 3pip install torch torchvision xformers --index-url https://download.pytorch.org/whl/cu118 4 5# Installing APEX 6pip install ninja 7git clone https://github.com/NVIDIA/apex 8cd apex 9pip install -v --disable-pip-version-check --no-cache-dir --no-build-isolation --global-option="--cpp_ext" --global-option="--cuda_ext" ./ 10cd .. 11 12# Installing Flash Attn 13pip install packaging 14pip install flash-attn --no-build-isolation 15 16# Clone the inference Repo 17git clone https://github.com/adarshxs/Capx-Llama3.1-Vision 18cd Capx-Llama3.1-Vision 19pip install -e .
• Run cli server:
  bash

  1python -m bunny.serve.cli \ 2 --model-path Capx/Llama-3.1-Vision \ 3 --model-type llama3.1-8b \ 4 --image-file /path/to/image \ 5 --conv-mode llama