You can deploy this model and start earning money today!

julius8787/
correctness-detection

safetensors

Views

No views yet

Correctness Detection Model

Model Description

This model is designed to analyze whether the assistant's responses in a dialogue are factually correct. The model evaluates the factual accuracy, logical consistency, and overall correctness of the assistant's responses. It generates a binary answer of YES or NO, indicating whether the assistant's response is factually correct.

The model is fine-tuned using LoRA (Low-Rank Adaptation) on the Meta-Llama-3-8B-Instruct base model, allowing for efficient adaptation and maintaining high performance.

Key features:

Analyzes the factual correctness of assistant responses.
Outputs a binary response (YES/NO) to indicate correctness.
Focuses on verifying facts, logic, and consistency of the assistant’s response.

Intended Use

Evaluating whether assistant-generated responses are factually accurate.
Ensuring the quality and reliability of AI-generated content.
Suitable for use in applications where the correctness of the information is critical.

How to Use

Here is an example of how to use the model for correctness detection:

python

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel, PeftConfig

# Base model and tokenizer
base_model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(base_model_name)

# Load the LoRA-adapted model
model = PeftModel.from_pretrained(base_model, "julius8787/correctness-detection")

# Define the dialogue
messages = [
    {"role": "system", "content": "You are a scientist whose only task is to analyze whether the assistant's responses in the dialogue are factually correct. Silently reason through the steps of analyzing correctness by verifying the facts, logic, and consistency of the response. After your analysis, only respond with YES or NO."},
    {"role": "user", "content": "What is the capital of the United States of America?"}, #TODO Change to your question
    {"role": "assistant", "content": "The capital of the United States of America is Washington D.C."} #TODO Change to your answer
]

# Prepare input for the model
input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

# Specify stop tokens for generation
terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]

# Generate the response
outputs = model.generate(
    input_ids,
    max_new_tokens=1,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.001,
    top_p=1,
)

# Decode the binary response (YES/NO)
response = outputs[0][input_ids.shape[-1]:]
print(tokenizer.decode(response, skip_special_tokens=True))