Universal-NER/UniNER-7B-all quantized to 4bit with GPTQ and stored with 1GB shard size.
The model
Universal-NER/UniNER-7B-all was quantized to 4bit, group_size 128, and act-order=True with auto-gptq integration in transformers (
https://huggingface.co/blog/gptq-integration).
1prompt_template = """A virtual assistant answers questions from a user based on the provided text.
2USER: Text: {input_text}
3ASSISTANT: I’ve read this text.
4USER: What describes {entity_name} in the text?
5ASSISTANT:
6"""
It is recommended to format input according to the prompt template mentioned above during inference for best results.
The model is small enough to be loaded in free-tier Colab with a T4 GPU:
https://gist.github.com/sebastianschramm/9903b2714e30d870d7e1e097c6b5c9e3The original full precision model and its associated data are released under the CC BY-NC 4.0 license. Hence, the same license applies for the 4bit version.
