You can deploy this model and start earning money today!

end765/
Endless-1b

Name: Endless-1b
Rating: 2.5 (5 reviews)

safetensors

gpt2

apache-2.0

Views

No views yet

Endless-1b

Model Description

Endless-1b is a decoder-only transformer language model trained with a three-stage approach: pretraining on diverse text corpora, supervised fine-tuning (SFT) on conversational data, and direct preference optimization (DPO) for alignment. The model architecture balances efficiency with decent performance on conversational and code generation tasks.

Model Details

Architecture

Model Type: Decoder-only Transformer
Parameters: 950 million (0.95B)
Architecture Dimensions:
- Hidden Size: 1,728
- Layers: 24
- Attention Heads: 16
- Feed-Forward Dimension: 6,912
- Context Length: 1,024 tokens
- Vocabulary Size: 50,257 (GPT-2 tokenizer)

Training Approach

The model was trained in three stages:

Stage 1: Pretraining

Trained from scratch on diverse text corpora
Datasets:
- FineWeb (web-scale dataset)
- Wikipedia
- Project Gutenberg
- Additional book collections
Hardware: Nvidia H100

Stage 2: Supervised Fine-Tuning (SFT)

Fine-tuned on high-quality conversational and instruction-following data
Datasets:
- LMSYS Chat conversations
- OpenOrca instruction dataset
- UltraChat conversational data
- Additional synthetic and freely available dialogue datasets
Total training tokens: Approximately 2 billion
Batch size: 128 (effective, with gradient accumulation)
Learning rate: 3e-6 with cosine decay and warmup
Hardware: Kaggle TPU v3-8

Stage 3: Direct Preference Optimization (DPO)

Aligned using preference pairs from UltraFeedback dataset
Filtered for English-only content and Python/JavaScript programming tasks
DPO beta: 0.1
Batch size: 32 (effective)
Learning rate: 5e-7 with cosine decay
Hardware: NVIDIA H100 GPU

Technical Optimizations

The H100 training implementation includes:

Flash Attention 3 for efficient self-attention computation
SwiGLU activation functions in feed-forward networks
RMSNorm for layer normalization
BFloat16 mixed precision training
Fused AdamW optimizer

Intended Use

Primary Use Cases

Conversational AI applications
Question answering systems
Content generation
Code assistance (Python and JavaScript)
Educational tutoring

Out-of-Scope Uses

Multilingual applications or non-English content generation
Code generation in languages other than Python and JavaScript (e.g., C++, Java, Rust, Go)
Advanced mathematical proofs, theorem proving, or symbolic computation
High-stakes decision making without human oversight
Generation of harmful, biased, or illegal content
Medical, legal, or financial advice

Training Data

Pretraining Data

FineWeb: Large-scale web corpus
Wikipedia: English Wikipedia dump
Project Gutenberg: Public domain books
Additional Books: Curated collection of educational and literary texts

SFT Training Data

LMSYS Chat: Real-world conversational data from ChatBot Arena
OpenOrca: High-quality instruction-following dataset
UltraChat: Multi-turn dialogue dataset
Synthetic Datasets: Generated conversational examples
Additional Open Datasets: Curated instruction and dialogue data from freely available sources

DPO Training Data

UltraFeedback: Preference pairs with chosen and rejected responses
Filtering Criteria:
- English language only (intentionally monolingual for quality)
- Programming content limited to Python and JavaScript
- Maximum sequence length of 900 tokens
- Excluded C++, Java, and other programming languages
- Excluded advanced mathematics and symbolic reasoning tasks

Rationale for Filtering: The model was intentionally optimized for high-quality English conversational capabilities rather than multilingual or broad technical coverage. By focusing exclusively on English and limiting programming support to Python and JavaScript, the model achieves better performance within its specialized domain. Advanced mathematics and non-English content were excluded to concentrate model capacity on conversational fluency and practical coding assistance in the most commonly used languages.

Data Preprocessing

All training data underwent rigorous filtering to remove:

Non-English content (intentional focus on English quality)
Excessively long sequences beyond model capacity
Low-quality or repetitive responses
Code examples in unsupported programming languages (C++, Java, Ruby, Go, etc.)
Advanced mathematical proofs and symbolic notation
Multilingual or code-switched content

Evaluation

Performance Characteristics

The model demonstrates medium performance on its target domains:

Single-turn English conversation coherence
Instruction following in English
Factual question answering in English
Basic mathematical reasoning and arithmetic at times

Design Philosophy: Rather than attempting broad coverage across all languages and programming paradigms, this model prioritizes depth over breadth. The intentional focus on English and popular programming languages (Python/JavaScript) allows for higher quality outputs within these domains compared to models with more diffuse training objectives.

Check out the paper i published on https://scaling-laws-limit.endless.sbs

Limitations

By Design (Intentional Constraints):

English-only optimization - not trained for multilingual tasks
Programming support limited to Python and JavaScript only
Mathematical reasoning focused on basic arithmetic and explanations, not advanced proofs

Technical Limitations:

Context window limited to 1,024 tokens
May exhibit biases present in training data
May produce hallucinations or factually incorrect information
Performance degrades on out-of-domain tasks (non-English, unsupported programming languages, advanced mathematics)

Usage

Installation

pip install transformers torch

Loading the Model

python

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "https://huggingface.co/end765/Endless-1b"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

Inference Example

python

from endless_chat import EndlessChat
chat = EndlessChat("end765/Endless-1b")

response = chat.generate("Hello!")
print(response)

# Multi-turn conversation
history = [
    {"role": "user", "content": "Hi"},
    {"role": "assistant", "content": "Hello!"}
]
response = chat.chat(history, "How are you?")
print(response)

Conversation Format

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

class EndlessChat:
    
    def __init__(self, model_name="end765/Endless-1b"):
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(model_name).to(self.device)
        self.model.eval()
        
        if self.tokenizer.pad_token is None:
            self.tokenizer.pad_token = self.tokenizer.eos_token
    
    def generate(self, prompt, max_tokens=256, temp=0.7):
        formatted = f"<|human|>{prompt}<|end|><|assistant|>"
        inputs = self.tokenizer(formatted, return_tensors="pt").to(self.device)
        
        with torch.inference_mode():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=max_tokens,
                temperature=temp,
                top_p=0.9,
                do_sample=True,
                pad_token_id=self.tokenizer.eos_token_id
            )
        
        response = self.tokenizer.decode(outputs[0], skip_special_tokens=False)
        response = response.split("<|assistant|>")[-1].split("<|end|>")[0].strip()
        return response
    
    def chat(self, history, message, max_tokens=256, temp=0.7):
        conversation = ""
        for turn in history:
            if turn["role"] == "user":
                conversation += f"<|human|>{turn['content']}<|end|>"
            else:
                conversation += f"<|assistant|>{turn['content']}<|end|>"
        
        conversation += f"<|human|>{message}<|end|><|assistant|>"
        inputs = self.tokenizer(conversation, return_tensors="pt").to(self.device)
        
        with torch.inference_mode():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=max_tokens,
                temperature=temp,
                top_p=0.9,
                do_sample=True,
                pad_token_id=self.tokenizer.eos_token_id
            )
        
        response = self.tokenizer.decode(outputs[0], skip_special_tokens=False)
        response = response.split("<|assistant|>")[-1].split("<|end|>")[0].strip()
        return response


chat = EndlessChat("end765/Endless-1b")

print(chat.generate("Hello!"))

history = []
while True:
    user_input = input("You: ").strip()
    if user_input.lower() in ['quit', 'exit']:
        break
    if not user_input:
        continue
    
    response = chat.chat(history, user_input) if history else chat.generate(user_input)
    print(f"Assistant: {response}")
    
    history.append({"role": "user", "content": user_input})
    history.append({"role": "assistant", "content": response})

Ethical Considerations

Bias and Fairness

The model may exhibit biases present in its training data, including but not limited to:

Cultural and demographic biases
Programming language preferences (Python/JavaScript)
English-centric worldview

Users should be aware of these limitations and implement appropriate safeguards when deploying the model.

Safety and Alignment

While the model underwent alignment training through DPO, it may still generate:

Factually incorrect information
Harmful or inappropriate content
Biased or stereotypical responses

Human oversight is recommended for production deployments, particularly in sensitive applications.

Environmental Impact

Carbon Footprint

Hardware:
- Kaggle TPU v3-8 (pretraining and SFT)
- NVIDIA H100 (DPO alignment)
Training Time: Approximately 2 weeks hours total across all stages.

The environmental impact was minimized through:

Efficient training techniques including mixed precision and gradient accumulation
Optimized attention mechanisms (Flash Attention 3)
Use of Kaggle's TPU infrastructure for compute-intensive stages

Citation

If you use this model in your research or applications, please cite:

bibtex

@misc{Endless-1b2026,
  author = {Endless},
  title = {TinyTransformer-950M: Efficient Conversational Language Model with DPO Alignment},
  year = {2026},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/end765/Endless-1b}},
}

License

This model is released under the Apache 2.0 License. See LICENSE file for details.

The model was trained on datasets with various licenses. Users should review and comply with the licenses of the underlying datasets:

Arena Human Preference Dataset
Tulu-3 SFT Personas
UltraFeedback

Acknowledgments

This model was trained using:

HuggingFace Transformers library
Flash Attention implementation
Kaggle TPU v3-8 infrastructure (pretraining and SFT stages)
NVIDIA H100 infrastructure (DPO alignment stage)

Special thanks to:

Kaggle for providing free TPU resources and some datasets
The creators of FineWeb, Wikipedia, Project Gutenberg, LMSYS, OpenOrca, UltraChat, and UltraFeedback datasets
The open-source community

Contact

For questions, issues, or collaboration inquiries, please open an issue on the model repository or contact mnbc80115@gmail.com.

Version History

v1.0 (Current)

Initial release
950M parameter model
Three-stage training: pretraining, SFT, and DPO alignment
Support for English conversational tasks and Python/JavaScript code generation.