not-for-all-audiences model-index:
name: Halu-OAS-8B-Llama3 results:
- task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics:
  - type: acc_norm value: 64.08 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Halu-OAS-8B-Llama3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics:
  - type: acc_norm value: 83.35 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Halu-OAS-8B-Llama3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics:
  - type: acc value: 67.8 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Halu-OAS-8B-Llama3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics:
  - type: mc2 value: 53.45 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Halu-OAS-8B-Llama3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics:
  - type: acc value: 79.79 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Halu-OAS-8B-Llama3 name: Open LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics:
  - type: acc value: 68.61 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=Hastagaras/Halu-OAS-8B-Llama3 name: Open LLM Leaderboard

This is an abliterated version of the HALU 8B Llama3 v0.3 model.

GGUF: Static/Imatrix made available by mradermacher

The orthogonal abliteration process was performed on Kaggle's 2xT4 instance in under 30 minutes.

The orthogonal abliteration process used in this model is based on the method created by wassname, utilizing the Baukit library. The original code can be found in this GitHub Gist.

A slightly modified version of the earlier version of the original code was used, which aimed to improve readability. The notebook used for the abliteration process can be found here.

The following are the benchmark results from the Chaiverse Leaderboard.

The difference in safety scores is 0.10 between the standard version and the OAS version. This means the orthogonalization method works despite using very few examples.

WARNING

This model has not been extensively tested or evaluated, and its performance characteristics are currently unknown. It may generate harmful, biased, or inappropriate content. Please exercise caution and use it at your own risk and discretion.

NOTES

The model's temperature setting influences its refusal to generate certain content. Higher temperature values increase refusal, while lower temperatures reduce refusal.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Avg.	69.51
AI2 Reasoning Challenge (25-Shot)	64.08
HellaSwag (10-Shot)	83.35
MMLU (5-Shot)	67.80
TruthfulQA (0-shot)	53.45
Winogrande (5-shot)	79.79
GSM8k (5-shot)	68.61

RichardErkhov/Hastagaras_-_Halu-OAS-8B-Llama3-8bits

Original model description:

This is an abliterated version of the HALU 8B Llama3 v0.3 model.

Open LLM Leaderboard Evaluation Results

RichardErkhov/
Hastagaras_-_Halu-OAS-8B-Llama3-8bits