You can deploy this model and start earning money today!

firdavsus/
LLM_350M_DENSE

Name: LLM_350M_DENSE
Availability: InStock

Views

No views yet

LLM_350M_DENSE

Model Description

LLM_350M_DENSE is a foundational, autoregressive, dense transformer large language model containing approximately 350 million parameters. This model was built, initialized, and trained from scratch using the custom architecture templates provided in the companion firdavsus/LLM_350M_DENSE GitHub repository.

The model design heavily draws optimization inspiration from contemporary architectures (such as Gemma/Llama-style layers), featuring advanced mechanisms like Multi-Query/Grouped-Query Attention, RoPE (Rotary Position Embeddings), and SwiGLU activation functions to maximize training stability and throughput efficiency on standard hardware environments.

Model Features & Specifications

Parameters: ~350,000,000 (350M)
Attention Mechanism: Dense Multi-Head/Grouped-Query Attention with FlashAttention-2 compilation bindings.
Context Length: 2048 / 4096 tokens (depending on exact positional execution adjustments).
Target Profiles: Multilingual text generation, light reasoning tasks, embedding generations, and low-latency edge deployment.

Architectural Details

The core implementation utilizes a decoupled token embedding system paired with the following configurations:

Hidden Dimension Size: Standard deep hidden layer projection size optimized for intermediate representation bandwidth.
Activation Layer: SwiGLU (Switching Gated Linear Unit) variants.
Normalization: RMSNorm applied pre-layer to maintain variance bounding throughout massive token pre-training loops.
Positional Layout: Rotary Position Embeddings (RoPE) applied to the query and key heads per-layer.

Intended Uses & Limitations

Target Applications

Resource-Constrained Environments: High-throughput inference loops on single customer-grade GPUs or local edge servers.
Architectural Benchmarking: Serving as a strong, predictable baseline for comparing structural modifications (e.g., Mixture-of-Experts vs. Dense variants).
Domain Adaptation: Highly suited for continued pre-training on localized languages like Uzbek (uz) or specific code structures.

Limitations

Parameter Capacity: Given its 350M parameter footprint, the model does not inherently exhibit zero-shot complex mathematical or deep logical multi-step execution paths common in massive 7B+ multi-billion parameter foundation models.
Tokenizer Alignment: Downstream prompt formats depend directly on the chosen vocabulary generation run (specified inside the training scripts in the GitHub repo).

Quickstart Inference

You can initialize and interface with the network components directly via PyTorch using the source scripts in the accompanying repository.

python

import torch
from model import Transformer, ModelArgs  # Imported from your firdavsus/LLM_350M_DENSE repository
from tokenizer import Tokenizer

# 1. Initialize architectural parameters
args = ModelArgs(
    dim=1024,
    n_layers=24,
    n_heads=16,
    vocab_size=32000,
    max_seq_len=2048
)

# 2. Build model and load checkpoint weights
device = "cuda" if torch.cuda.is_available() else "cpu"
model = Transformer(args).to(device)

checkpoint = torch.load("path_to_model_checkpoint.pt", map_location=device)
model.load_state_dict(checkpoint["model"])
model.eval()

print("Model architecture compiled and weights successfully loaded from disk.")