aixsim-200M is a transformer-based language model with approximately 200 million parameters (embedding layer params excluded). It uses RMSNorm for normalization and is trained on the FineWeb-Edu dataset.

Developed by: AICrossSim
Funded by: ARIA
Model type: Transformer Language Model
Language(s) (NLP): English
Tokenizer: HuggingFaceTB/cosmo2-tokenizer
Repository: AICrossSim/NewComputeBench

Training Details

Experiment setup and training logs can be found at wandb run.

Usage

python

import transformers

model_name="AICrossSim/clm-200m"
model = transformers.AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)

lm-evaluation-harness

Tasks	Version	Filter	Metric		Value		Stderr
wikitext	2	none	bits_per_byte	↓	1.0994	±	N/A
		none	byte_perplexity	↓	2.1427	±	N/A
		none	word_perplexity	↓	58.8531	±	N/A