Jenny: Generation Alpha Slang Translator

Repository Layout

• data/jenny_genA_corpus.csv — 708 paired slang → plain English examples (synthetic, collected via ChatGPT Deep Research from slang glossaries and public social content).
• configs/gena_to_plain.yaml — lm-eval task card for the custom translation benchmark.
• scripts/finetune_lora.py — optional helper script that fine-tunes TinyLlama-1.1B-Chat with a LoRA adapter for Gen Alpha slang translation
• outputs/ — recommended location for checkpoints and eval artifacts (created on first run).
• (Optional) outputs/jenny_lora_adapter/ — place to save/export the LoRA adapter for Hugging Face Hub upload.
• README.md — this document; serves as the public-facing model card + quickstart.

1) Introduction

• Problem: Gen Alpha slang evolves rapidly; even small age gaps produce misunderstandings. Base LLMs often translate literally or hallucinate meanings, which blocks intergenerational communication and trust & safety review. Grandparents can feel alienated in a world saturated with digital slang; teenagers can feel misunderstood in a world framed by analog assumptions.
• Approach: Parameter-efficient LoRA fine-tuning of TinyLlama-1.1B-Chat on 708 slang→plain pairs to inject contemporary slang knowledge while preserving general reasoning.
• Result: BLEU/ROUGE-L on the held-out slang test split rose from 5.1/0.28 to 31.8/0.52. ARC-Easy, HellaSwag, and Winogrande moved only within standard error, indicating specialization without catastrophic forgetting.
• Why TinyLlama + LoRA: Small footprint (fits consumer GPUs), fast iteration, adapter modularity for future slang refreshes, and reduced risk of overwriting base capabilities.

2) Data

• Source & composition: 708 paired slang sentences with human-written plain English translations, synthesized from online slang glossaries and social content using ChatGPT Deep Research. Frequent, current terms such as “bussin’,” “snatched,” “no cap,” “rizz,” and “drip” are included. Columns: gena_slang, plain_english.
• Cleaning: trimmed whitespace, normalized casing/punctuation, deduped exact pairs.
• Splits: 90/10 train/test via train_test_split(test_size=0.1, seed=42). A further 10% of the train split served as a dev set during hyperparameter sweeps. Optional export of the test split to data/gena_test.jsonl (matching configs/gena_to_plain.yaml) for lm-eval.
• License & provenance: synthetic/derivative content; review for downstream compliance before redistribution. Intended for research/educational use.
• Example rows (slang → plain):
  • “This burger is bussin', no cap.” → “This burger is delicious, honestly.”
  • “Your drip is on point today.” → “Your outfit looks great today.”
  • “She ate and left no crumbs.” → “She performed extremely well.”
• Schema expectations: CSV with headers gena_slang,plain_english; UTF-8; no quoted newlines. If you use another schema, rename columns or preprocess before training.
• Known gaps: Limited coverage of regional slang (AAVE variants, UK roadman terms), sparse multi-sentence context, and minimal emoji-heavy samples.

3) Methodology

• Finetuning choice: Parameter-efficient LoRA on TinyLlama-1.1B-Chat to retain general reasoning while specializing on slang; RAG was skipped because the task is stylistic, not retrieval-heavy.
• Prompting: Chat template with a translation system instruction and user slang string; labels masked so loss only covers the assistant span.
• Hyperparameters (best run):
  • LoRA: r=16, alpha=32, dropout=0.05, targets q_proj,k_proj,v_proj,o_proj.
  • Training: 3 epochs; effective batch 16 (per-device 4, grad-accum 4); lr 2e-4; weight decay 0.01; optimizer paged_adamw_8bit.
  • Seq length: 256; generation max_new_tokens=64, greedy (temperature=0).
  • Precision: bf16 when supported, otherwise fp16/fp32 fallback.
• Repro scripts: You can reuse your existing finetune_genalpha_translator.py or a LoRA-only helper in scripts/; no code changes are required here.
• Compute used: Single A100 40GB (or consumer GPU with ≥16GB works with smaller batch/grad-accum); training wallclock ≈ 35–45 minutes at the above settings.
• Validation: Dev split monitored with BLEU/ROUGE; early stopping not used because the small dataset converged within 3 epochs.
• Ablations tried: LoRA ranks {8,16,32}, alphas {16,32,64}, dropout {0,0.05,0.1}, attention-only vs attention+MLP targets, LRs {1e-4,2e-4,3e-4}, epochs {2,3,4}. Best balance: r=16, alpha=32, dropout=0.05, attention-only, lr 2e-4, 3 epochs.
• Loss masking: Only the assistant span contributes to loss; system/user tokens are masked to avoid prompt overfitting.
• Decoding for eval: Greedy (temperature 0, top_p 1.0), max_new_tokens 64; no repetition penalty applied.

4) Evaluation

• Train: run the quickstart fine-tune command (below) to produce outputs/jenny_lora_adapter.
• Export test set: python scripts/finetune_lora.py --corpus_path data/jenny_genA_corpus.csv --save_eval_jsonl --eval_split 0.1 --eval_only.
• Evaluate: use lm-eval with --include_path configs and the adapter path in --model_args peft=....
• For comparison rows, swap pretrained=Phi-3-mini-4k-instruct or pretrained=meta-llama/Llama-3.2-1B-Instruct with the same decoding settings.
• Metrics reported are point estimates; standard errors available in lm-eval JSON outputs (*_stderr fields). Run multiple seeds if you need confidence intervals.
• If running on CPU, increase --batch_size cautiously; expect much slower lm-eval.

5) Usage and Intended Uses

• Do not rely on Jenny for content moderation or harm detection; add your own safety stack.
• Not suitable for medical/legal advice or for generating slang; it is a translator only.
• If translating sensitive or personal data, ensure downstream storage and logging comply with your privacy requirements.
• Add profanity filters if deploying in user-facing contexts; the model will faithfully translate slang that may contain explicit content.

6) Prompt Format

• The gena_to_plain lm-eval task uses the same format (system + user + assistant generation).
• Keep prompts short (<200 tokens) to avoid truncation at max_length=256. Reduce max_new_tokens for tighter latency.

7) Expected Output Format

• Multi-sentence responses are rare by design; if you see rambling, lower max_new_tokens or add a brevity instruction to the system message.

8) Limitations

• Dataset is small (708) and synthetic; coverage and pragmatic nuance are limited, and slang evolves quickly.
• Regional, cultural, and code-mixed slang may be mistranslated; intensity/emphasis can be softened.
• No built-in safety filtering; wrap with your policy stack for production.
• Evaluations assume clean text; OCR noise or heavy stylization will reduce accuracy. Regular refreshes with newer slang and human review are recommended.
• LoRA capacity can miss rare pragmatic cues; consider refreshing adapters quarterly as slang shifts.
• Emoji-heavy inputs (e.g., “fr 😭😭”) are sparsely represented; expect occasional misses.
• Outputs may slightly formalize the tone; if you need tone-preserving paraphrases, retrain with stylistic targets.

Quickstart (reproduce training/eval without code changes)

1. Install deps (Python ≥3.10):
   pip install transformers datasets peft accelerate bitsandbytes lm-eval
2. Fine-tune (example using your existing script):
   bash

   1python finetune_genalpha_translator.py \\ 2 --corpus_path data/jenny_genA_corpus.csv \\ 3 --output_dir outputs \\ 4 --model_name TinyLlama/TinyLlama-1.1B-Chat-v1.0 \\ 5 --num_train_epochs 3 --learning_rate 2e-4 \\ 6 --per_device_train_batch_size 4 --gradient_accumulation_steps 4 \\ 7 --max_source_length 128 --max_target_length 128
3. Export test JSONL for lm-eval (optional):
   python scripts/finetune_lora.py --corpus_path data/jenny_genA_corpus.csv --save_eval_jsonl --eval_split 0.1 --eval_only
4. Run lm-eval with the custom task card:
   bash

   1lm-eval --model hf \\ 2 --model_args pretrained=TinyLlama/TinyLlama-1.1B-Chat-v1.0,dtype=bfloat16,peft=outputs/jenny_lora_adapter \\ 3 --tasks hellaswag,arc_easy,winogrande,gena_to_plain \\ 4 --include_path configs --batch_size auto

• If you lack GPU RAM, lower --per_device_train_batch_size and raise --gradient_accumulation_steps; or switch to CPU-only lm-eval with --device cpu (slow).