You can deploy this model and start earning money today!

iZELX1/
llm-wikipedia

Name: llm-wikipedia
Availability: InStock

Views

No views yet

Wikipedia-trained Phi-2 (Merged)

This is a fine-tuned version of Microsoft's Phi-2 model, adapted for Wikipedia-style content generation. The LoRA weights have been merged into the base model for easier inference.

Model Details

Base Model: microsoft/phi-2
Fine-tuning Method: LoRA (Low-Rank Adaptation) with weights merged
Training Data: Wikipedia articles
Training Objective: Text generation and completion

Usage

With Transformers

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model
model = AutoModelForCausalLM.from_pretrained(
    "iZELX1/llm-wikipedia",
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)

# Load tokenizer
tokenizer = AutoTokenizer.from_pretrained("iZELX1/llm-wikipedia")

# Generate text
inputs = tokenizer("The history of artificial intelligence", return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.7)
text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(text)

Training Details

LoRA Rank: 64
LoRA Alpha: 16
Target Modules: ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"]
Training Steps: 2,184
Batch Size: 4
Learning Rate: 2e-4

Performance

Perplexity: ~12.5 (on validation set)
BLEU Score: ~0.15
ROUGE-1 F1: ~0.35

Limitations

This is a personal project for educational purposes. The model may:

Generate factually incorrect information
Exhibit biases present in the training data
Produce inappropriate content
Have limited knowledge outside of Wikipedia-style content

License

MIT License - see the LICENSE file for details.

Acknowledgments

Microsoft for the Phi-2 base model
Hugging Face for the transformers library
The PEFT library for LoRA implementation