An experimental 85.6M-parameter encoder-decoder model for turning a conversation's first user message into a short title.
Quick start
This checkpoint uses a custom MLX architecture and is not a drop-in AutoModel checkpoint. Download the complete repository so the implementation, custom kernels, configuration, tokenizer, and weights remain together:
Use modeling_titan_mlx.py and kernels.py from the downloaded snapshot. The training-time decode configuration was greedy generation with a minimum length of 3, no repeated token unigrams, and a maximum of 32 tokens. The repository does not yet include a packaged generation CLI, so loading and decoding require custom integration.
Results
TitleBench (full 21,448-example split)
Metric
Value
ROUGE-1
0.152
ROUGE-2
0.025
ROUGE-L
0.151
chrF
0.082
BLEU-4
0.284
Exact match and repetition rate were not reported for this run. The evaluation used the repository's official TitleBench scorer with greedy decoding, min_len=3, unigram repetition blocking, and a 32-token maximum.
Takeaway: This release documents an architecture experiment, not a state-of-the-art checkpoint. Its title-generation quality trails NeroT-29M on the directly comparable overlap metrics.
Architecture
Component
Configuration
Parameters
~85.6M with tied embeddings
Model type
Bidirectional encoder + autoregressive decoder
Encoder / decoder layers
20 / 6
Hidden size
512
Attention
8 query heads, 2 key/value heads, head size 64
FFN
SwiGLU, size 1,408
Position encoding
RoPE, base 10,000
Normalization
RMSNorm, epsilon 1e-6
Vocabulary
16,385 tokens
Maximum source / target lengths
512 / 48
Embeddings
Tied input/output embeddings
Dropout
0.0
Training story
1. Interleaved pre-training
MASS-style span corruption over lmsys/lmsys-chat-1m conversations.
Approximately 15% masking with one dedicated mask token.
Chat-title pairs were upweighted 2× and represented about 45% of batches.
2. Title-only fine-tuning
Title pairs only.
Peak learning rate 8e-5 with cosine decay.
Data processing and deduplication used seed 42.
Data lineage
Source
Role
Recorded volume
j0no12/chat-titles-unified
Real title pairs
223,853 train pairs
Reconstructed BananaMind-Chat-Title-200K
Real/teacher title pairs reconstructed by gated LMSYS ordinal ID
127,335 train pairs
BananaMind/BananaMind-Title-1.0 over fresh LMSYS first messages
Self-distilled titles
248,132 prompts
lmsys/lmsys-chat-1m
Span-corruption corpus
~1M conversations
All 21,448 normalized TitleBench inputs, including locally reconstructed gated rows, were excluded before encoding. The pipeline also performed exact-pair deduplication and input-key priority deduplication with real examples preferred over synthetic examples.
Evaluation details
Dataset: j0no12/TitleBench, all 21,448 examples.
Decode: greedy, minimum length 3, no repeated unigrams, maximum 32 tokens.
Scorer: TitleBench's published scorer.
Missing metrics: exact match and repetition were not recorded in the original result table and are intentionally not inferred here.
Reproducibility
Item
Value
Model repository revision documented by this card update
74667d68b8edb28729674b1c0f4b85256293b63e
Data-processing seed
42
Implementation
modeling_titan_mlx.py + kernels.py
Configuration
config.json
Weights
model.safetensors
Tokenizer
BPE, vocab 16,385; pad 0, EOS 1, BOS 3, mask 16,384
Training hardware
Apple M5 Max
The repository does not publish the complete training command, environment lockfile, or processed-data hashes. Reproduction is therefore architectural rather than bit-for-bit.
Limitations
Experimental checkpoint with substantially lower TitleBench overlap scores than NeroT-29M.
Custom MLX code is required; generic Transformers loading is not implemented despite the repository's historical library_name metadata.
The data is English-dominant even though some underlying LMSYS conversations may be multilingual.
Synthetic targets inherit the teacher model's style and errors.
The benchmark guards against direct normalized-input overlap, but that does not prove absence of all semantic near-duplicates or upstream pretraining exposure.
Hardware and framework
Trained from scratch with MLX on an Apple M5 Max.
Custom encoder-decoder implementation included in the repository.
Teacher attribution: BananaMind/BananaMind-Title-1.0, based on LiquidAI's LFM2.5-350M under the LFM Open License v1.0.
Files
File
Description
model.safetensors
Released model weights
config.json
Exact released architecture configuration
modeling_titan_mlx.py
MLX model implementation
kernels.py
Supporting MLX kernels
tokenizer.json
BPE tokenizer
README.md
This model card
Citation
bibtex
1@misc{nerot86mexp,
2 author = {j0no12},
3 title = {NeroT-86M-Exp: An Experimental Encoder-Decoder for Chat Titles},
4 year = {2026},
5 howpublished = {\url{https://huggingface.co/j0no12/NeroT-86M-Exp}}
6}
License
CC-BY-4.0 for the original model weights and repository material. Attribution is required. Gated LMSYS data was used locally under its own terms and is not redistributed as raw conversation text here. Upstream data and teacher-model terms remain applicable.
