This is a LoRA (Low-Rank Adaptation) adapter trained on a tiny random LLaMA model. The model and adapter are intentionally small and produce random outputs—they are not meant for any real inference tasks. The primary purpose is to provide a lightweight adapter for testing deployment pipelines, inference servers, and LoRA loading mechanisms.

Model type: LoRA adapter for causal language modeling
Language(s): English
License: MIT
Finetuned from: hmellor/tiny-random-LlamaForCausalLM

Model Sources

Repository: syaffers/tiny-random-llama-lora (training code)

Uses

Direct Use

This adapter is intended for:

Smoke testing LoRA adapter loading in inference pipelines
Testing deployment configurations with minimal resource usage
Validating HuggingFace PEFT integration in your infrastructure

Out-of-Scope Use

This model should not be used for:

Any real text generation or NLP tasks
Production applications
Any use case requiring meaningful outputs

How to Get Started with the Model

python

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model = AutoModelForCausalLM.from_pretrained("hmellor/tiny-random-LlamaForCausalLM")
model = PeftModel.from_pretrained(base_model, "syaffers/tiny-random-llama-lora")
tokenizer = AutoTokenizer.from_pretrained("syaffers/tiny-random-llama-lora")

# Generate (output will be random/meaningless)
inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=10)
print(tokenizer.decode(outputs[0]))

Training Details

Training Data

iamholmes/tiny-imdb - A tiny subset of IMDB reviews used for quick training iterations.

Training Procedure

Training Hyperparameters

Training regime: fp32
Batch size: 4
Learning rate: 1e-4
Epochs: 3
Warmup steps: 10
Max sequence length: 128

LoRA Configuration

Parameter	Value
r (rank)	8
lora_alpha	16
target_modules	q_proj, v_proj
lora_dropout	0.05
bias	none
task_type	CAUSAL_LM

Technical Specifications

Model Architecture and Objective

LoRA adapter applied to the query and value projection layers of a tiny random LLaMA architecture for causal language modeling.

Compute Infrastructure

Hardware

Apple M3 Pro (36GB unified memory)
macOS Sequoia 15.6.1

Software

Transformers
PEFT 0.18.0
PyTorch
Datasets

Framework versions

PEFT 0.18.0