Views
No views yet
sciencebase-metadata-llama3-70b (v 1.0)| Field | Value |
|---|---|
| Developed by | Quan Quy, Travis Ping, Tudor Garbulet, Chirag Shah, Austin Aguilar |
| Contact | quyqm@ornl.gov • pingts@ornl.gov • garbuletvt@ornl.gov • shahch@ornl.gov • aguilaral@ornl.gov |
| Funded by | U.S. Geological Survey (USGS) & Oak Ridge National Laboratory – ARM Data Center |
| Model type | Autoregressive LLM, instruction-tuned for structured → metadata generation |
| Base model | meta-llama/Llama-3.1-70B-Instruct |
| Languages | English |
| Finetuned from | unsloth/Meta-Llama-3.1-70B-Instruct |
| Resource | Link |
|---|---|
| Repository | https://huggingface.co/ARM-Development/Llama-3.3-70B-tabular-1.0 |
| Demo | https://colab.research.google.com/drive/1saCEFhkBYDhQWkdTwnwiE_-AiWmD6p0f#scrollTo=WeniLP-Ah1QL |
| Hyper-parameter | Value |
|---|---|
| Max sequence length | 20 000 |
| Precision | fp16 / bf16 (auto) |
| Quantisation | 4-bit QLoRA (load_in_4bit=True) |
| LoRA rank / α | 16 / 16 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Optimiser | adamw_8bit |
| LR / schedule | 2 × 10⁻⁴, linear |
| Epochs | 1 |
| Effective batch | 4 (1 GPU × grad-acc 4) |
| Trainer | trl SFTTrainer + peft 0.15.2 |
| Field | Value |
|---|---|
| GPU | 1 × NVIDIA A100 80 GB |
| Total training hours | ~120 hours |
| Cloud/HPC provider | ARM Cumulus HPC |
| Package | Version |
|---|---|
| Python | 3.12.9 |
| PyTorch | 2.6.0 + CUDA 12.4 |
| Transformers | 4.51.3 |
| Accelerate | 1.6.0 |
| PEFT | 0.15.2 |
| Unsloth | 2025.3.19 |
| BitsAndBytes | 0.45.5 |
| TRL | 0.15.2 |
| Xformers | 0.0.29.post3 |
| Datasets | 3.5.0 |
| … |
Llama-3.1-70B-Instruct; causal-LM objective with structured-to-text instruction prompts.