You can deploy this model and start earning money today!

sbhikha/
Gemma9B_Inkuba_LoRa_adapter_v1

Name: Gemma9B_Inkuba_LoRa_adapter_v1
Availability: InStock

Views

No views yet

Model Card for Model ID

This model card contains a LoRa adapter for the Gemma-9b-it model. It aims to improve Machine Translation performance into South African langauges.

Model Details

Model Description

The LoRa adapter was trained on a subset of the lelapa/Inkuba-instruct dataset targeting the langauges English, isiZulu and isiXhosa. The data was sampled from the Inkuba-instruct dataset using langauge-ids.

Model Sources [optional]

Developed by: Sandil Bhikha
Model type: LoRa adapter
Language(s) (NLP): English, isiZulu, isiXhosa
Finetuned from model: Gemma-9b-it

Paper [optional]: [More Information Needed]
Demo [optional]: [More Information Needed]

Uses

Aims to improve Machine Translation performance on South African langauges. Based on Gemma-9b-it, so the model is generally task adaptable.

Bias, Risks, and Limitations

The datatset was sampled from lelapa/Inkuba-instruct dataset. Minor inconsistencies were present in translations from the training data.

How to Get Started with the Model

Use the code below to get started with the model.

base_model_id = "google/gemma-2-9b-it"
adapter_model_id = "sbhikha/Gemma9B_Inkuba_double_quant"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)

double_quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config = double_quant_config,
    device_map="auto"
)

config = PeftConfig.from_pretrained(adapter_model_id)
peft_model = PeftModel.from_pretrained(model, adapter_model_id)

text = "Translate this from English to isiZulu: Sawubona, unjani?"

input_ids = tokenizer(text), return_tensors="pt").to("cuda:0")

outputs = peft_model.generate(**input_ids, max_new_tokens=100)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

English

sbhikha/Inkuba_english_train_sliced_v1
sbhikha/Inkuba_english_dev_sliced_v1

Zulu

sbhikha/Inkuba_isizulu_train_sliced_v1
sbhikha/Inkuba_isizulu_dev_sliced_v1

Xhosa

sbhikha/Inkuba_xhosa_train_sliced_v1
sbhikha/Inkuba_xhosa_dev_sliced_v1

Training Hyperparameters

Training regime: bf16 mixed precision

LoRa Config:

Rank = 64
Alpha = 32
Target Modules (Linear Modules): ['q_proj', 'gate_proj', 'up_proj', 'k_proj', 'v_proj', 'o_proj', 'down_proj']
Dropout = 0.05

Training Arguments:

Learning Rate: 2e-4
per_device_train_batch_size: 1
gradient_accumulation_steps: 4
warmup_steps = 3
max_steps = 100
optim="paged_adamw_8bit"

Evaluation

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: 1x H100 GPU
Hours used: 23.7
Cloud Provider: Lambda Labs
Compute Region: Utah, USA
Carbon Emitted: [More Information Needed]

sbhikha/Gemma9B_Inkuba_LoRa_adapter_v1