You can deploy this model and start earning money today!

TaoTern/
TaoNet-pico-A2-pretrain

Name: TaoNet-pico-A2-pretrain
Availability: InStock

Views

No views yet

TaoNet Pico A2 Pretrain

Model summary

TaoNet Pico A2 Pretrain is a pretrained TaoNet A2 variant. It was trained as an optimized model for perplexity-filter based dataset preprocessing and model benchmarking, not as a general-purpose assistant or instruction-following model. This model uses the TaoNet MLA + RoPE architecture. It was trained on TaoData dataset, built for high-quality language modeling in small models. The training objective was standard language modeling.

Generation

Transformers

python

import torch
from transformers import AutoModelForCausalLM

model_id = "Lobakkang/TaoNet-pico-A2-pretrain"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = torch.bfloat16 if device == "cuda" else torch.float32

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device)
model.eval()

Sample inference

python

import sys
from pathlib import Path

import torch
from transformers import AutoModelForCausalLM

ROOT = Path(__file__).resolve().parent
SRC = ROOT / "src"
if str(SRC) not in sys.path:
    sys.path.insert(0, str(SRC))

from taoTrain.inference.loading import load_tokenizer

MODEL_ID = "Lobakkang/TaoNet-pico-A2-pretrain"
TOKENIZER_PATH = ROOT / "tokenizer" / "tokenizer.model"

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
dtype = torch.bfloat16 if device.type == "cuda" else torch.float32

tokenizer = load_tokenizer(TOKENIZER_PATH)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    trust_remote_code=True,
    torch_dtype=dtype,
).to(device)
model.eval()

prompt = "Hello world"
input_ids = tokenizer.encode(prompt, return_tensors="pt").to(device)

with torch.no_grad():
    output_ids = model.generate(
        input_ids=input_ids,
        max_new_tokens=32,
        do_sample=True,
        temperature=0.7,
        top_p=0.95,
    )

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))

Benchmarks

These results are from the pretrained checkpoint:

Task	Score
MMLU	0.2295
HellaSwag	0.2725
ARC Easy	0.3695
ARC Challenge	0.2227
PIQA	0.5517
Winogrande	0.5083

Limitations

This checkpoint is best treated as a pretrained research model rather than a normal-purpose application model.

It was trained for perplexity-filter style preprocessing and filtering workflows.
It is not tuned for chat, instruction following, or tool use.
Output quality is expected to vary outside the data and evaluation distribution used during pretraining.
As with other small language models, it can produce plausible but incorrect text.
The model should be validated before any downstream deployment or product use.

Training

Model

Architecture: TaoNet / MLA + RoPE
Tokenizer: SentencePiece
Framework: TaoTrain
Dataset: TaoData

Hardware

GPU: 1 x RTX 5090

Software

Training framework: TaoTrain

License

MIT License.

Citation

bibtex

@software{taonet_pico_a2_pretrain,
  title={TaoNet Pico A2 Pretrain},
  author={Lobakkang},
  year={2026},
  url={https://huggingface.co/Lobakkang/TaoNet-pico-A2-pretrain}
}

If you use TaoTrain or TaoData in a writeup, cite the corresponding project references in addition to this checkpoint.

TaoTern/TaoNet-pico-A2-pretrain