This model is NOT suitable for legal advice or production use. It hallucinates case citations, confuses legal concepts, and produces factually incorrect information. This is an experimental model for educational purposes only.

Model Description

This is Qwen2.5-0.5B fine-tuned on a company law textbook through continued pretraining. The model learned legal vocabulary and writing style but exhibits significant factual inaccuracies and overfitting to the training data's structure.

Developed by: Pactoria
Model type: Causal Language Model (Decoder-only Transformer)
Language: English
License: Apache 2.0
Base Model: Qwen/Qwen2.5-0.5B
Training Approach: Full fine-tuning with FP16 mixed precision
Domain: UK Company Law

Training Details

Training Data

Source: "Company Law" textbook by Smith and Keenan (14th edition)
Size: 2.3 MB plain text (550,382 tokens)
Content: Definitions, case law, Companies Act 2006 references, exam questions
Processing: Chunked into 1,073 examples of 1,024 tokens each with 50% overlap
Total Training Tokens: 1,098,752 (across 3 epochs)

Training Procedure

Hyperparameters:

Epochs: 3
Batch size: 1 per device
Gradient accumulation: 8 steps
Effective batch size: 8
Learning rate: 5e-5
LR scheduler: Cosine with 3% warmup
Optimizer: AdamW (weight_decay=0.01)
Max gradient norm: 1.0
Sequence length: 1,024 tokens
Training regime: FP32 → FP16 mixed precision

Infrastructure:

GPU: NVIDIA Tesla T4 (16GB)
Training time: 20 minutes
Total steps: 405
Framework: HuggingFace Transformers 4.52.4

Loss Trajectory:

Step 1:   2.303
Step 100: 1.874
Step 200: 0.980
Step 300: 0.465
Step 400: 0.237
Final:    1.035 (average)

Intended Use

Primary Use Case

Educational demonstration of continued pretraining pipeline. This model is part of a three-phase learning experiment:

Phase 1: Continued pretraining on domain text ✓ (this model)
Phase 2: Supervised fine-tuning on Q&A pairs (planned)
Phase 3: DPO alignment (planned)

How to Use

python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "your-username/qwen-0.5b-law-phase1",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(
    "your-username/qwen-0.5b-law-phase1",
    trust_remote_code=True
)

# Generate (greedy decoding for stability)
prompt = "A corporation is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Performance & Limitations

What Works

✅ Generates grammatically correct legal text
✅ Uses appropriate legal terminology
✅ References Companies Act 2006 and case names
✅ Maintains coherent sentence structure
✅ No catastrophic collapse (no repetition/gibberish)

Critical Limitations

❌ Hallucinates case citations - Invents fake cases and misattributes legal principles
❌ Confuses case law - Misrepresents what famous cases established
❌ Overfitted to textbook format - Outputs exam questions instead of explanations
❌ Factual inaccuracies - Provides incorrect legal information
❌ No fact-checking - Cannot distinguish real from invented law

Example Hallucinations

Prompt: "The doctrine of separate legal personality"
Output: "was developed by... Robert Goff and Sir Henry Price in... Bell v Lever Bros Ltd [1920]"
Reality: ❌ Completely false. The doctrine comes from Salomon v Salomon [1897], not Bell v Lever Bros.

Prompt: "In Macaura v Northern Assurance"
Output: "shareholders are entitled to the dividends..."
Reality: ❌ Wrong. Macaura is about insurable interest, not dividends.

Out-of-Scope Uses

DO NOT use this model for:

Legal advice or consultation
Academic citations or legal research
Court filings or legal documents
Any situation where accuracy matters
Production applications
Decision-making about legal matters

Misuse risks:

Providing harmful legal misinformation
Making incorrect legal decisions
Academic misconduct if used for assignments
Liability issues if used for actual legal work

Bias & Ethical Considerations

Training bias: Single textbook from UK jurisdiction (not representative of all legal systems)
Temporal bias: Training data reflects law as of textbook publication (may be outdated)
Accessibility bias: Written in formal legal English (may exclude non-experts)
Hallucination risk: Model confidently states false information (dangerous in legal context)

Evaluation Results

Manual Testing (35 prompts across 8 categories)

Test Category	Result
Legal definitions	Coherent but sometimes wrong
Case law citations	Frequently hallucinates
Companies Act references	Better accuracy on Section 42
Question answering	Mixed; sometimes outputs exam questions
Long-form generation	Coherent but factually unreliable
Edge cases	Handles unusual inputs without crashing

Overall Assessment: Model learned legal writing style but not factual accuracy.

Environmental Impact

Hardware: 1x NVIDIA Tesla T4 (16GB)
Training time: 20 minutes
Cloud provider: Kaggle
Estimated CO2: ~0.02 kg (20 min × 70W GPU × 0.475 kg CO2/kWh grid average)

Technical Specifications

Model Architecture

Type: Decoder-only Transformer (Qwen 2.5 architecture)
Parameters: 494,032,768 (all trainable)
Hidden size: 896
Layers: 24
Attention heads: 14
Vocab size: 151,665 tokens
Context length: 131,072 tokens (training used 1,024)

Software

Transformers: 4.52.4
PyTorch: 2.6.0+cu124
CUDA: 12.4
Python: 3.11.13

Citation

If you use this model for research on continued pretraining or domain adaptation:

bibtex

@misc{qwen-law-phase1,
  author = {Your Name},
  title = {Qwen 0.5B Company Law Phase 1: Continued Pretraining Experiment},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/your-username/qwen-0.5b-law-phase1}},
  note = {Experimental model - not for legal use}
}

Acknowledgments

Base model: Qwen Team
Training framework: HuggingFace Transformers
Infrastructure: Kaggle (free T4 GPU access)

Model Card Authors

[Your Name] - Experimental continued pretraining for educational purposes

Contact

For questions about the training methodology (not legal questions):

HuggingFace: @your-username
Issues: Model repository

Version: 1.0 (Phase 1 only)
Last Updated: October 2025
Status: Experimental - Educational use only


This model card is honest about limitations while still documenting what was accomplished. Replace `your-username` and `[Your Name]` with your actual details before uploading.

rzeraat/qwen-0.5b-law-phase1

Qwen 0.5B - Company Law (Phase 1: Continued Pretraining)

⚠️ Critical Disclaimer