You can deploy this model and start earning money today!

naver/
pisco-llama

Name: pisco-llama
Availability: InStock

safetensors

COCOM

custom_code

meta-llama/Llama-3.1-8B-Instruct

finetune

cc-by-4.0

Views

No views yet

Model Card for PISCO-llama

PISCO is a context compression model to be used for efficient inference when doing Retrieval Augmented Generation (RAG), particularly optimized for question answering.

PISCO contains two adapters around a backbone LLM:

An encoder adapter trained to perform compression of input contexts (the retrieved documents in RAG) into a set of 8 embedding vectors
A decoder adapter, which can take as input sets of embeddings vectors from documents and a query and provide an answer

With a compressed collection of documents to retrieve from, inference becomes about x5 faster. PISCO models have very small loss in accuracy on a wide set of QA benchmarks (0-3%).

Developed by: Naver Labs Europe
License: CC BY-NC 4.0.

Model: Pisco-llama
Backbone model: meta-llama/Llama-3.1-8B-Instruct
Model size: 8.11 billion parameters
Compression rate: x16: each document (of size up to 128 tokens) is converted into 8 embedding vectors.

Usage

python

from transformers import AutoModel

pisco = AutoModel.from_pretrained('naver/pisco-llama').to('cuda')

# Example documents and question:
documents = [
    [
        "Weldenia is a monotypic genus of flowering plant in the family Commelinaceae, first describ ed in 1829. It has one single species: Weldenia candida, which grows originally in Mexico and Guatemala.",
        "Hagsatera is a genus of flowering plants from the orchid family, Orchidaceae. There are two known species, native to Mexico and Guatemala",
        "Alsobia is a genus of flowering plants in the family Gesneriaceae, native to Mexico, Guatemala and Costa Rica. The two species are succulent, stoloniferous herbs and were previously included in the genus \"Episcia\". Recent molecular studies have supported the separation of \"Alsobia\" from \"Episcia\""
    ]
]

questions = ["Which genus of plant grows originally in Mexico and Guatemala, Phylica or Weldenia?"]

# End-to-end usage
out = pisco.generate_from_text(questions=questions, documents=documents, max_new_tokens=64)
print('Generated answer', out)

# Document compression:
embeddings = pisco.compress_documents(documents=documents[0])

# Generation from compressed documents:
out = pisco.generate_from_compressed_documents_and_questions(questions=questions, compressed_documents=embeddings)

The recommended usage is to provide documents cropped to about 128 tokens, which is common practice when doing RAG.

Model features

PISCO enables high accuracy responses from the compressed documents
PISCO is robust to various domains We tested its compression/decoding abilities on various sets of data.
PISCO enables x5 faster generation when the collection documents to retrieve from is pre-compressed.

License

This work is licensed under CC BY-NC 4.0.

Cite

TODO

Acknowledgements

Model trained at Naver Labs Europe
Team: