This model was developed to improve text generation accuracy and contextual understanding for specific domains covered in the provided PDF assets. It leverages the underlying architecture of gpt2 and is optimized for the structure and vocabulary found in document-based datasets.

Key Specifications

Attribute	Value
Base Architecture	gpt2
Format	PyTorch (Transformers)
Task	Causal Language Modeling
Language	English (en)

Training Configuration

The model was fine-tuned using the following high-level hyperparameters to ensure stability and convergence:

Epochs: 3
Batch Size: 1
Learning Rate: 5e-05
Optimized for: Cross-Entropy Loss
Hardware: cuda

Performance Metrics

Total Training Loss: 3.680997848510742
Training Runtime: 3.6704 seconds

Data Preprocessing

The training data was sourced from dummy.pdf. The pipeline included:

Extraction: Text recovery using pypdf.
Normalization: Regex-based whitespace cleaning and token normalization.
Tokenization: Model-specific subword tokenization with a max sequence length of 512 tokens.

Usage Instructions

To utilize this model for inference, use the following snippet:

python

from transformers import AutoTokenizer, AutoModelForCausalLM

checkpoint = "singtan/my-llm-finetuned-pdf"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)

prompt = "Insert your context here"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=150)
print(tokenizer.decode(outputs[0]))

Contact

Developed by Bibek - Senior AI Engineering Portfolio.