en
pt datasets:
cnmoro/WizardVicuna-PTBR-Instruct-Clean model-index:
name: Mistral-7B-Portuguese results:
- task: type: text-generation name: Text Generation dataset: name: ENEM Challenge (No Images) type: eduagarcia/enem_challenge split: train args: num_few_shot: 3 metrics:
  - type: acc value: 58.08 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: BLUEX (No Images) type: eduagarcia-temp/BLUEX_without_images split: train args: num_few_shot: 3 metrics:
  - type: acc value: 48.68 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: OAB Exams type: eduagarcia/oab_exams split: train args: num_few_shot: 3 metrics:
  - type: acc value: 37.08 name: accuracy source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: Assin2 RTE type: assin2 split: test args: num_few_shot: 15 metrics:
  - type: f1_macro value: 90.31 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: Assin2 STS type: eduagarcia/portuguese_benchmark split: test args: num_few_shot: 15 metrics:
  - type: pearson value: 76.55 name: pearson source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: FaQuAD NLI type: ruanchaves/faquad-nli split: test args: num_few_shot: 15 metrics:
  - type: f1_macro value: 58.84 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: HateBR Binary type: ruanchaves/hatebr split: test args: num_few_shot: 25 metrics:
  - type: f1_macro value: 79.21 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: PT Hate Speech Binary type: hate_speech_portuguese split: test args: num_few_shot: 25 metrics:
  - type: f1_macro value: 68.87 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard
- task: type: text-generation name: Text Generation dataset: name: tweetSentBR type: eduagarcia-temp/tweetsentbr split: test args: num_few_shot: 25 metrics:
  - type: f1_macro value: 64.71 name: f1-macro source: url: https://huggingface.co/spaces/eduagarcia/open_pt_llm_leaderboard?query=cnmoro/Mistral-7B-Portuguese name: Open Portuguese LLM Leaderboard

This is a finetuned version of mistralai/Mistral-7B-Instruct-v0.2 using unsloth on a instruct portuguese dataset, as an attempt to improve the performance of the model on the language.

No benchmarks have been executed yet.

The original prompt format was used:

<s>[INST] {Prompt goes here} [/INST]

Open Portuguese LLM Leaderboard Evaluation Results

Detailed results can be found here

Metric	Value
Average	64.7
ENEM Challenge (No Images)	58.08
BLUEX (No Images)	48.68
OAB Exams	37.08
Assin2 RTE	90.31
Assin2 STS	76.55
FaQuAD NLI	58.84
HateBR Binary	79.21
PT Hate Speech Binary	68.87
tweetSentBR	64.71

RichardErkhov/cnmoro_-_Mistral-7B-Portuguese-8bits

Original model description:

Open Portuguese LLM Leaderboard Evaluation Results

RichardErkhov/
cnmoro_-_Mistral-7B-Portuguese-8bits