You can deploy this model and start earning money today!

juliushase/
verbosity-detection

Name: verbosity-detection
Availability: InStock

safetensors

Views

No views yet

Verbosity Detection Model

Model Description

This model is designed to evaluate whether a given assistant response in a dialogue is verbose or concise. It generates a binary response of YES or NO, indicating if the assistant's reply contains more words than necessary to convey the relevant information. The model analyzes the response based on its relevance, clarity, and conciseness.

The model is fine-tuned with LoRA (Low-Rank Adaptation) on the Meta-Llama-3-8B-Instruct base model, optimizing it for efficiency while leveraging the capabilities of large language models.

Key aspects:

Evaluates verbosity of assistant responses.
Binary output (YES/NO) for verbose or concise evaluation.
The analysis is based on relevance, clarity, and conciseness.

Intended Use

Identifying whether an assistant's response is unnecessarily verbose.
Analyzing the clarity and relevance of assistant responses.
Enhancing conversational AI by improving response efficiency.

How to Use

Here is an example of how to use the model for verbosity detection:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

# Base model and tokenizer
base_model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_name) # Make sure you use a token to access meta/llama
base_model = AutoModelForCausalLM.from_pretrained(base_model_name)

# Load the LoRA-adapted model
model = PeftModel.from_pretrained(base_model, "juliushase/verbosity-detection")

# Define the dialogue
messages = [
    {"role": "system", "content": "You are a scientist whose sole task is to determine whether the assistant's response in the dialogue is verbose, meaning it contains more words than necessary to convey the relevant information. Your analysis should consider the response's relevance, clarity, and conciseness. After reasoning through your analysis, respond only with YES if the response is verbose or NO if it is concise."},
    {"role": "user", "content": "Question"},
    {"role": "assistant", "content": "Answer"}
]

# Prepare input for the model
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Specify stop tokens for generation
terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

# Generate the response
outputs = model.generate(
    input_ids,
    max_new_tokens=1,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.001,
    top_p=1,
)

# Decode the binary response (YES/NO)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))