You can deploy this model and start earning money today!

RichardErkhov/
HuggingFaceTB_-_SmolLM-135M-gguf

Name: HuggingFaceTB_-_SmolLM-135M-gguf
Rating: 2.5 (77 reviews)

endpoints_compatible

gguf

template

Views

No views yet

Quantization made by Richard Erkhov.

Github

Discord

Request more models

SmolLM-135M - GGUF

Model creator: https://huggingface.co/HuggingFaceTB/
Original model: https://huggingface.co/HuggingFaceTB/SmolLM-135M/

Name	Quant method	Size
SmolLM-135M.Q2_K.gguf	Q2_K	0.08GB
SmolLM-135M.IQ3_XS.gguf	IQ3_XS	0.08GB
SmolLM-135M.IQ3_S.gguf	IQ3_S	0.08GB
SmolLM-135M.Q3_K_S.gguf	Q3_K_S	0.08GB
SmolLM-135M.IQ3_M.gguf	IQ3_M	0.08GB
SmolLM-135M.Q3_K.gguf	Q3_K	0.09GB
SmolLM-135M.Q3_K_M.gguf	Q3_K_M	0.09GB
SmolLM-135M.Q3_K_L.gguf	Q3_K_L	0.09GB
SmolLM-135M.IQ4_XS.gguf	IQ4_XS	0.09GB
SmolLM-135M.Q4_0.gguf	Q4_0	0.09GB
SmolLM-135M.IQ4_NL.gguf	IQ4_NL	0.09GB
SmolLM-135M.Q4_K_S.gguf	Q4_K_S	0.1GB
SmolLM-135M.Q4_K.gguf	Q4_K	0.1GB
SmolLM-135M.Q4_K_M.gguf	Q4_K_M	0.1GB
SmolLM-135M.Q4_1.gguf	Q4_1	0.09GB
SmolLM-135M.Q5_0.gguf	Q5_0	0.1GB
SmolLM-135M.Q5_K_S.gguf	Q5_K_S	0.1GB
SmolLM-135M.Q5_K.gguf	Q5_K	0.1GB
SmolLM-135M.Q5_K_M.gguf	Q5_K_M	0.1GB
SmolLM-135M.Q5_1.gguf	Q5_1	0.1GB
SmolLM-135M.Q6_K.gguf	Q6_K	0.13GB
SmolLM-135M.Q8_0.gguf	Q8_0	0.13GB

Original model description:

library_name: transformers license: apache-2.0 language:

en datasets:
HuggingFaceTB/smollm-corpus

SmolLM

Model Summary
Limitations
Training
License
Citation

Model Summary

SmolLM is a series of state-of-the-art small language models available in three sizes: 135M, 360M, and 1.7B parameters. These models are built on Cosmo-Corpus, a meticulously curated high-quality training dataset. Cosmo-Corpus includes Cosmopedia v2 (28B tokens of synthetic textbooks and stories generated by Mixtral), Python-Edu (4B tokens of educational Python samples from The Stack), and FineWeb-Edu (220B tokens of deduplicated educational web samples from FineWeb). SmolLM models have shown promising results when compared to other models in their size categories across various benchmarks testing common sense reasoning and world knowledge. For detailed information on training, benchmarks and performance, please refer to our full blog post.

This is the SmolLM-135M

Generation

pip install transformers

Running the model on CPU/GPU/multi GPU

Using full precision

python

# pip install transformers
from transformers import AutoModelForCausalLM, AutoTokenizer
checkpoint = "HuggingFaceTB/SmolLM-135M"
device = "cuda" # for GPU usage or "cpu" for CPU usage
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for multiple GPUs install accelerate and do `model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto")`
model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)
inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to(device)
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

bash

>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
Memory footprint: 12624.81 MB

Using torch.bfloat16

python

# pip install accelerate
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
checkpoint = "HuggingFaceTB/SmolLM-135M"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
# for fp16 use `torch_dtype=torch.float16` instead
model = AutoModelForCausalLM.from_pretrained(checkpoint, device_map="auto", torch_dtype=torch.bfloat16)
inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

bash

>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
Memory footprint: 269.03 MB

Quantized Versions through `bitsandbytes`

Using 8-bit precision (int8)

python

# pip install bitsandbytes accelerate
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
# to use 4bit use `load_in_4bit=True` instead
quantization_config = BitsAndBytesConfig(load_in_8bit=True)
checkpoint = "HuggingFaceTB/SmolLM-135M"
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint, quantization_config=quantization_config)
inputs = tokenizer.encode("def print_hello_world():", return_tensors="pt").to("cuda")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

bash

>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
# load_in_8bit
Memory footprint: 162.87 MB
# load_in_4bit
>>> print(f"Memory footprint: {model.get_memory_footprint() / 1e6:.2f} MB")
Memory footprint: 109.78 MB

Limitations

While SmolLM models have been trained on a diverse dataset including educational content and synthetic texts, they have limitations. The models primarily understand and generate content in English. They can produce text on a variety of topics, but the generated content may not always be factually accurate, logically consistent, or free from biases present in the training data. These models should be used as assistive tools rather than definitive sources of information. Users should always verify important information and critically evaluate any generated content. For a more comprehensive discussion of the models' capabilities and limitations, please refer to our full blog post..

This repository contains a converted version of our latest trained model. We've noticed a small performance difference between this converted checkpoint (transformers) and the original (nanotron). We're currently working to resolve this issue.

Training

Model

Architecture: For architecture detail, see the blog post.
Pretraining steps: 600k
Pretraining tokens: 600B
Precision: bfloat16
Tokenizer: HuggingFaceTB/cosmo2-tokenizer

Hardware

GPUs: 64 H100

Software

Training Framework: Nanotron

License

Apache 2.0

Citation

bash

@misc{allal2024SmolLM,
      title={SmolLM - blazingly fast and remarkably powerful}, 
      author={Loubna Ben Allal and Anton Lozhkov and Elie Bakouch and Leandro von Werra and Thomas Wolf},
      year={2024},
}

RichardErkhov/HuggingFaceTB_-_SmolLM-135M-gguf

Original model description:

SmolLM

Table of Contents

Model Summary

Generation

Running the model on CPU/GPU/multi GPU

Quantized Versions through bitsandbytes

Limitations

Training

Model

Hardware

Software

License

Citation

RichardErkhov/
HuggingFaceTB_-_SmolLM-135M-gguf

Quantized Versions through `bitsandbytes`