You can deploy this model and start earning money today!

ISTA-MLCV/
llama_2_7b_aside

Name: llama_2_7b_aside
Availability: InStock

transformers

safetensors

llama

text-generation-inference

endpoints_compatible

Views

No views yet

Llama 2 7B ASIDE

This is the Llama 2 7B model augmented and fine-tuned with the ASIDE modification, trained and evaluated in the paper ASIDE: Architectural Separation of Instructions and Data in Language Models.

Model Description

ASIDE applies an orthogonal rotation to the embeddings of data tokens, thus creating clearly distinct representations of instructions and data tokens without introducing any additional parameters.

Usage

To use this model, first clone and follow the installation instructions in the official ASIDE Repository.

Inside the repository, run the following code snippet (also provided here as a script) to do inference with this model.

python

import torch
import deepspeed
import json
import os
from huggingface_hub import login

from model_api import CustomModelHandler  # Import your custom handler
from model_api import format_prompt  # Import your prompt formatting function

# Define your instruction and data
instruction_text = "Translate to German."
data_text = "Who is Albert Einstein?"

# Model configuration
hf_token = os.environ["HUGGINGFACE_HUB_TOKEN"]
login(token=hf_token)
embedding_type = "forward_rot"  
base_model = "meta-llama/Llama-2-7b-hf"
model_path = "Embeddings-Collab/llama_2_7b_forward_rot_emb_SFTv110_from_base_run_6"

# Initialize the model handler
handler = CustomModelHandler(
    model_path, 
    base_model, 
    base_model, 
    model_path, 
    None,
    0, 
    embedding_type=embedding_type, 
    load_from_checkpoint=True
)

# Initialize DeepSpeed inference engine
engine = deepspeed.init_inference(
    model=handler.model,
    mp_size=torch.cuda.device_count(),  # Number of GPUs
    dtype=torch.float16,
    replace_method='auto',
    replace_with_kernel_inject=False
)
handler.model = engine.module

# Load prompt templates
with open("./data/prompt_templates.json", "r") as f:
    templates = json.load(f)

template = templates[0]  
instruction_text = format_prompt(instruction_text, template, "system")
data_text = format_prompt(data_text, template, "user")

# Generate output
output, inp = handler.call_model_api_batch([instruction_text], [data_text])
print(output)

Citation

If you use this model, please cite our paper:

@inproceedings{
  zverev2026aside,
  title={{ASIDE}}: Architectural Separation of Instructions and Data in Language Models},
  author={Egor Zverev and Evgenii Kortukov and Alexander Panfilov and Alexandra Volkova and Rush Tabesh and Sebastian Lapuschkin and Wojciech Samek and Christoph H. Lampert},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=C81TnwHiRM}
}

ISTA-MLCV/llama_2_7b_aside

Llama 2 7B ASIDE

Model Description

Usage

Citation

ISTA-MLCV/
llama_2_7b_aside