This is a conversion of talkie-lm/talkie-web-13b-base to the HuggingFace Transformers format. The original model was distributed as a raw PyTorch checkpoint with a custom inference library; this version can be loaded directly with AutoModelForCausalLM and AutoTokenizer.

[!NOTE] This base model was ported to Hugging Face format by Xiaoning Wang, based on the version automatically converted by Hugging Face's ML Intern at lewtun/talkie-1930-13b-it-hf — an AI agent for ML engineering tasks. Try it yourself via the CLI or the Demo.

Model Summary
How to Use
Architecture Details
Conversion Notes
License

Model Summary

talkie-web-13b-it is a 13B-parameter instruction-tuned language model from the talkie family, developed by Alec Radford, Nick Levine, and David Duvenaud.

Key Features

Vintage knowledge: trained exclusively on pre-1931 text, offering a unique window into early 20th-century language and thought
Instruction-tuned: fine-tuned for conversational use with a simple chat template
13B parameters in bfloat16 (~26 GB VRAM)
2048 token context window

How to Use

Installation

This model uses custom modeling code. Make sure you have a recent version of transformers installed:

pip install -U transformers torch

Basic Generation

To be added.

Multi-turn Chat

To be added.

Chat Template

There is no chat template for this base model.

Architecture Details

talkie is a 40-layer decoder-only GPT with several distinctive architectural choices:

Component	Details
Parameters	13B
Layers	40
Attention heads	40 (MHA, no GQA)
Hidden size	5120
Head dimension	128
Intermediate size (MLP)	13696
Position encoding	RoPE (θ = 1,000,000)
Activation	SwiGLU
Normalization	RMSNorm (pre-norm)
Context length	2048
Vocabulary	65,540 (65,535 BPE + 5 special tokens)
Precision	bfloat16

Notable architectural features:

QK-normalization: RMSNorm is applied to queries and keys after RoPE
Per-head gain: learnable scalar gain per attention head, applied to queries
Embedding skip connections: each transformer block receives a residual connection from the (normalized) input embeddings
Activation gains: learnable scalar gains on attention and MLP residual streams (initialized to (2·L)^(-0.5))
lm_head weight gain: a learnable scalar applied to the output projection weights

Conversion Notes

This model was converted from the original talkie-lm/talkie-web-13b-it PyTorch checkpoint using the reference talkie codebase as ground truth. The conversion involved:

Model weights: the .pt state dict was remapped to a PreTrainedModel subclass (TalkieForCausalLM) and saved as safetensors
Tokenizer: the tiktoken BPE vocabulary was converted to a PreTrainedTokenizerFast with the HuggingFace TikTokenConverter, including all 5 special tokens (<|endoftext|>, <|end|>, <|user|>, <|assistant|>, <|system|>)
Validation: logits were compared on 4 test prompts covering chat, system prompts, and raw completion — all top-5 decoded tokens match exactly, with cosine similarity ≥ 0.99999994

Since this is a custom architecture, loading requires trust_remote_code=True.

License

Apache 2.0 — same as the original model.

xiaoningwang/talkie-web-13b-base-hf

talkie-web-13b-it (Transformers format)

Table of Contents