from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")
model = AutoModelForCausalLM.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")

# Generate text
prompt = "The process of photosynthesis"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        temperature=0.8,
        top_k=50,
        do_sample=True,
        pad_token_id=tokenizer.eos_token_id
    )

generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(generated_text)

Custom Generation (using original model code)

python

# If you want to use the original model implementation
from model import ModelConfig, LlamaModel
from transformers import AutoTokenizer
import torch

tokenizer = AutoTokenizer.from_pretrained("Saiteja23/smollm2-135m-fineweb-edu2.7BT")
# Load using custom model class - see repository for details

Model Performance

This model is designed for:

Text Continuation: Completing partial sentences or paragraphs
Educational Content: Generating educational explanations and content
Creative Writing: Short-form creative text generation
General Language Modeling: Basic language understanding and generation

Limitations

No Instruction Following: This is a pretrained model without instruction tuning
No Safety Training: No specific safety or alignment training applied
Educational Bias: Training data bias toward educational content
Size Limitations: 135M parameters limit complex reasoning capabilities

Training Infrastructure

Hardware: Single A100 80GB GPU
Framework: PyTorch with custom training loop
Distributed Training: Single GPU
Mixed Precision: bfloat16

Citation

If you use this model, please cite:

bibtex

@misc{smollm2-135m-fineweb-edu,
  title={SmolLM2-135M trained on FineWeb-edu},
  author={Your Name},
  year={2024},
  howpublished={\url{https://huggingface.co/Saiteja23/smollm2-135m-fineweb-edu2.7BT}},
}

Acknowledgments

Base architecture inspired by SmolLM2 (HuggingFace)
Training data from FineWeb-edu dataset
Built with PyTorch and Hugging Face Transformers

This model was trained as part of a language model pretraining experiment. Use responsibly and be aware of potential biases in generated content.