You can deploy this model and start earning money today!

CHwensX/
llama-chinese-35m

Name: llama-chinese-35m
Availability: InStock

safetensors

llama

Views

No views yet

Llama-Chinese-35M

English | 中文

English

Model Description

A 35M-parameter Chinese LLaMA-style language model trained from scratch on Chinese Wikipedia data. This is an educational project demonstrating the complete pipeline of training a modern decoder-only Transformer.

Model Architecture

Architecture: LLaMA-style Decoder-Only Transformer
Layers: 5 transformer blocks
Hidden Size: 512
Attention Heads: 8 (no GQA)
Head Dimension: 64
FFN: SwiGLU (intermediate size: 1792)
Position Encoding: RoPE (theta=10000.0)
Normalization: RMSNorm (Pre-LN, eps=1e-6)
Weight Tying: lm_head.weight = embed_tokens.weight
Vocabulary Size: 32,000
Max Sequence Length: 512

Training Details

Training Data: ~18MB Chinese Wikipedia
Epochs: 3
Batch Size: 8
Learning Rate: 3e-4
Optimizer: AdamW
Hardware: Apple M5 (MPS)
Training Time: ~26 minutes

Usage

python

import torch
from modeling_llama import LlamaModel

# Load config
config = {
    "vocab_size": 32000,
    "embed_dim": 512,
    "num_layers": 5,
    "num_heads": 8,
    "num_kv_heads": 8,
    "ffn_dim": 1792,
    "context_length": 512,
    "pad_token_id": 0,
}

# Initialize model
model = LlamaModel(**config)

# Load weights
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

# Generate text
input_ids = torch.tensor([[2, 1, 2, 3, 4]])  # example input
with torch.no_grad():
    logits = model(input_ids)

Tokenizer

Type: ByteLevel BPE
Vocabulary Size: 32,000
Special Tokens: <pad>, <unk>, <bos>, <eos>

Load the tokenizer:

python

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

Files

model.safetensors - Model weights in safetensors format
config.json - Model configuration
modeling_llama.py - Model architecture code
tokenizer.json - Tokenizer vocabulary and merges
tokenizer_config.json - Tokenizer configuration
special_tokens_map.json - Special token mappings

中文

模型简介

从零训练的 35M 参数中文 LLaMA 风格语言模型，使用中文维基百科数据训练。这是一个教育项目，展示了训练现代 Decoder-Only Transformer 的完整流程。

模型架构

架构: LLaMA 风格 Decoder-Only Transformer
层数: 5 个 Transformer Block
隐藏维度: 512
注意力头: 8（无 GQA）
头维度: 64
前馈网络: SwiGLU（中间维度: 1792）
位置编码: RoPE（theta=10000.0）
归一化: RMSNorm（Pre-LN, eps=1e-6）
权重绑定: lm_head.weight = embed_tokens.weight
词表大小: 32,000
最大序列长度: 512

训练详情

训练数据: ~18MB 中文维基百科
训练轮数: 3
批次大小: 8
学习率: 3e-4
优化器: AdamW
硬件: Apple M5 (MPS)
训练时间: 约 26 分钟

使用方法

python

import torch
from modeling_llama import LlamaModel

# 加载配置
config = {
    "vocab_size": 32000,
    "embed_dim": 512,
    "num_layers": 5,
    "num_heads": 8,
    "num_kv_heads": 8,
    "ffn_dim": 1792,
    "context_length": 512,
    "pad_token_id": 0,
}

# 初始化模型
model = LlamaModel(**config)

# 加载权重
from safetensors.torch import load_file
state_dict = load_file("model.safetensors")
model.load_state_dict(state_dict)
model.eval()

# 生成文本
input_ids = torch.tensor([[2, 1, 2, 3, 4]])  # 示例输入
with torch.no_grad():
    logits = model(input_ids)

分词器

类型: ByteLevel BPE
词表大小: 32,000
特殊标记: <pad>, <unk>, <bos>, <eos>

加载分词器：

python

from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("tokenizer.json")

文件说明

model.safetensors - 模型权重（safetensors 格式）
config.json - 模型配置
modeling_llama.py - 模型架构代码
tokenizer.json - 分词器词表和合并规则
tokenizer_config.json - 分词器配置
special_tokens_map.json - 特殊标记映射

CHwensX/llama-chinese-35m

Llama-Chinese-35M

English

Model Description

Model Architecture

Training Details

Usage

Tokenizer

Files

中文

模型简介

模型架构

训练详情

使用方法

分词器

文件说明

CHwensX/
llama-chinese-35m