You can deploy this model and start earning money today!

ZK0622/
a0.1_mistral_adalora_sft

Name: a0.1_mistral_adalora_sft
Availability: InStock

peft

safetensors

1910.09700

ZK0622/mistral7b-legal-summarizer-adalora-v0.1

adapter

Views

No views yet

Model Card for Model ID

Model Details

Model Description

Mistral 7b finetuned via 2 stage fine tunning (DAPT & SFT) with Hierarchical Merging and PEFT method (AdaLoRA) on legal documents for the purpose of Legal Docs Summarization

Developed by: [ZK]
Model type: [Decoder Only]
Finetuned from model: [Mistral 7b]

Model Sources

Repository: [mistralai/Mistral-7B-v0.3]
Paper: [https://arxiv.org/pdf/2310.06825]

Training Details

Contact Owner for training details.

Training Data

[Only Data for SFT available "ZK0622/SFT"]

Evaluation

Testing Data, Factors & Metrics

Testing Data

[TO-BE-UPDATED]

Factors

[TO-BE-UPDATED]

Metrics

[TO-BE-UPDATED]

Results

PERPLEXITY

Model	Perplexity (Lower Better)
Mistral 7B Instruct v0.2	4.29
AdaLoRA Mistral 7B	(This Model) 3.73
LoRA Mistral 7B	3.54

ROUGE

Transformer Architecture	Model Name	Rouge-1	Rouge-2	Rouge-L	Rouge-L Sum
Encoder	Legal BERT	0.374	0.185	0.218	0.218
Encoder-Decoder	Legal LED	0.306	0.113	0.169	0.169
Encoder-Decoder	Legal T5	0.268	0.084	0.167	0.167
Decoder	Mistral 7B Base	0.440	0.195	0.269	0.273
Decoder (This Model)	AdaLoRA Mistral 7B	0.506	0.294	0.350	0.351
Decoder	LoRA Mistral 7B	0.564	0.376	0.437	0.437

Transformer Architecture	Model Name	Precision	Recall	BERT-F1
Encoder	Legal BERT	0.800	0.857	0.827
Encoder-Decoder	Legal LED	0.798	0.845	0.821
Encoder-Decoder	Legal T5	0.674	0.820	0.739
Decoder	Mistral 7B Base	0.852	0.866	0.858
Decoder (This Model)	AdaLoRA Mistral 7B	0.874	0.884	0.878
Decoder	LoRA Mistral 7B	0.908	0.890	0.898

Summary

Framework versions

PEFT 0.10.0
Transformer 4.40
DeepSpeed config: ZeRO Stage 2

ZK0622/a0.1_mistral_adalora_sft

Model Card for Model ID

Model Details

Model Description

Model Sources

Training Details

Training Data

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Framework versions

ZK0622/
a0.1_mistral_adalora_sft