You can deploy this model and start earning money today!

ManmohanSharma/
samosachaat-d24

Name: samosachaat-d24
Availability: InStock

ManmohanSharma/nanochat-d24-training-data

mit

Views

No views yet

samosaChaat —d24

A 1.38B-parameter conversational language model trained from scratch, with native web-search and calculator tool use, an explicit thinking mode, and a 16K token context window.

Small, spicy, and entirely open — built on 8× NVIDIA H100 GPUs in a few hours by Manmohan Sharma. Weights, tokenizer, training data, and scripts are all here.

Live demo: samosachaat.art

Model card


Architecture	nanochat GPT — transformer decoder, 24 layers, 12 heads, d_model=1536, head_dim=128
Parameters	1,384,122,122 (1.38 B)
Context length	16,384 tokens
Attention	SSSL pattern (3× sliding-window + 1× full attention per 4 layers) + rotary position embeddings
Training dtype	FP8 (tensorwise) on H100, bf16 compute
Tokenizer	rustbpe, vocab 32,768 (Karpathy nanochat tokenizer)
Hardware	8× NVIDIA H100 SXM HBM3 (Prime Intellect / Hyperbolic), Flash Attention 3
Base repo	Fork of karpathy/nanochat at manmohan659/nanochat

What it can do

Chat in natural English with persistent identity (knows it is samosaChaat, created by Manmohan Sharma)
Web search via the web_search tool, powered by Tavily — invoked automatically when a question asks about current / time-sensitive information
Calculator via the calculator tool with a restricted expression evaluator (percent, emi, cagr, compound interest, basic arithmetic)
Explicit reasoning mode — when prompted, produces <think>...</think> chain-of-thought traces, then a final answer
Indian-cuisine domain knowledge — trained on a curated desserts/street-food corpus (samosa chaat, rasgulla, biryani, etc.)

Repository layout

Path	Contents
`base_checkpoints/d24/`	Original base pretrain (step 5,568, val_bpb 0.718)
`base_checkpoints/d24-cpt/`	Continued pretrain with Nemotron + domain data (step 10,000, val_bpb 0.365)
`base_checkpoints/d24-cpt-16k/`	16K context extension (step 1,200, val_bpb 0.526 @ 16K seq)
`chatsft_checkpoints/d24-sft-r6/`	Production SFT — recommended (step 754, val_bpb 0.263, 97% probe pass rate)
`chatsft_checkpoints/d24-sft-r5/`	SFT round 5 (broader chat data, 91% pass)
`chatsft_checkpoints/d24-sft-r4/`	SFT round 4 (focused identity/domain, 93% pass)
`tokenizer/`	`tokenizer.pkl` + `token_bytes.pt` (vocab 32,768)
`datasets/`	Identity conversations, desserts Q&A, tool-use conversations
`scripts/`	Training pipeline scripts (see Training Report)
`evals/`	Probe-suite results per SFT round

The companion dataset repository ManmohanSharma/nanochat-d24-training-data hosts the 40 parquet shards (~18 GB) used for base pretraining and continued pretraining.

Quick usage

Hit the live endpoint

bash

curl -N -X POST https://manmohan659--samosachaat-inference-inference-generate.modal.run \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [
      {"role": "user", "content": "You are samosaChaat, a helpful AI assistant. Answer directly.\n\nWho created you?"}
    ],
    "temperature": 0.3,
    "max_tokens": 256
  }'

Returns a Server-Sent Events stream. Each data: {"token": "..."} line is one output token.

Load locally with nanochat

bash

git clone https://github.com/manmohan659/nanochat
cd nanochat
pip install -r requirements.txt

python

import torch
from nanochat.checkpoint_manager import load_model
from nanochat.engine import Engine
from nanochat.tools import build_default_tool_registry

# Download the weights first:
# from huggingface_hub import snapshot_download
# snapshot_download("ManmohanSharma/nanochat-d24",
#                   allow_patterns=["chatsft_checkpoints/d24-sft-r6/*", "tokenizer/*"])

device = torch.device("cuda")
model, tokenizer, meta = load_model(
    "sft", device, "eval", model_tag="d24-sft-r6", step=754
)
engine = Engine(model, tokenizer, tools=build_default_tool_registry())

# Direct chat
messages = [{"role": "user", "content":
    "You are samosaChaat, a helpful AI assistant. Answer directly and concisely.\n\n"
    "What is samosa chaat?"}]
tokens = tokenizer.render_for_completion({"messages": messages + [{"role":"assistant","content":""}]})
out, _ = engine.generate_batch(tokens, num_samples=1, max_tokens=200, temperature=0.3)
print(tokenizer.decode(out[0]))

Enable web search

Set TAVILY_API_KEY in the environment. build_default_tool_registry() auto-detects it and swaps from the mock search backend to the real Tavily backend. Without the key, tool calls return mock results.

System prompt conventions

Two modes, distinguished purely by the system prompt (the model was trained on both):

Direct mode (concise answers):

You are samosaChaat, a helpful AI assistant. Answer directly and concisely.

Thinking mode (visible chain-of-thought):

You are samosaChaat, a helpful AI assistant. Think step by step inside <think>...</think> tags, then give your final answer.

Tool-aware prompt (encourages web_search / calculator use):

You are samosaChaat, a helpful AI assistant with access to tools. Use web_search for facts that may change over time or require current information, and calculator for arithmetic. Otherwise answer directly.

All system prompts are merged into the first user message at tokenisation time, matching the nanochat tokenizer convention.

Evaluation

Final probe suite (33 prompts across 9 categories, PASS/FAIL grading — see evals/eval_results_v2.jsonl):

Category	d24-sft-r6
Factual recall (Paris, Au, 1945, speed of light, etc.)	6/6 (100%)
Indian cuisine / culture (samosa chaat, rasgulla, biryani, rupee, Taj Mahal, Diwali)	6/6 (100%)
Math (linear equations, percentages, rate problems)	4/4 (100%)
Identity (name, creator attribution, parameter count, rejects "are you ChatGPT?")	6/6 (100%)
Creative writing (haiku, limerick)	2/2 (100%)
General chat (intros, technical explanations, language compare)	3/3 (100%)
Tool use (web_search for current events, calculator for tips)	3/3 (100%)
Reasoning (day-of-week, multiplication, trick puzzles)	2/3 (67%)
Overall	32/33 (97%)

Per-round trajectory:

Round	Val bpb	Probe pass	Key change
Base pretrain (d24, step 5568)	0.72	—	Karpathy-style pretrain on ClimbMix
CPT (d24-cpt, step 10000)	0.365	—	Continued pretrain + domain data, 2K ctx
16K extension (d24-cpt-16k, step 1200)	0.526†	57% (pre-SFT base)	rope precompute @ 16K, short warmdown
SFT r1	0.401	79%	First chat SFT
SFT r2–r3	0.400	64–71%	Heavy creator upsample regressed chat quality
SFT r4	0.269	93%	Focused SFT, dropped noisy breadth data
SFT r5	0.261	91%	Merged with curated broad chat data
SFT r6	0.263	97%	Reasoning reinforcement

†val_bpb at 16K context is naturally higher than at 2K — the model is conditioning on longer sequences with more surprise.

See TRAINING_REPORT.md for the full pipeline.

Limitations

Numeric trivia hallucination — like all ~1 B parameter models, specific figures (GDP, population) may be wrong. The built-in web_search tool mitigates this when invoked.
Temporal reasoning — day-of-week arithmetic and similar multi-step temporal problems still miss.
Joint think-and-tool use — the SFT data trained <think> reasoning and <|python_start|> tool calling as separate patterns. When a user explicitly turns on thinking mode, the model sometimes reasons from memory rather than calling web_search. Turning off thinking mode (or using the tool-aware system prompt without the <think> clause) reliably invokes the search tool.
Knowledge cut-off — pretraining data cut-off is inherited from the base corpora. For current information always rely on web_search.

Safety and intended use

This model is released as an open educational artifact to demonstrate that a capable conversational LLM can be trained from scratch on modest academic-scale compute. It is not intended as a reference factual source — treat all outputs as draft information to be verified.

Credits

Built by Manmohan Sharma — AI Researcher and Full Stack Engineer, pursuing an MS in Computer Science (AI) at the University of San Francisco.

Base architecture: Andrej Karpathy's nanochat — MIT licensed
Base pretraining corpus: ClimbMix
Continued pretraining: Nemotron-Specialized, Nemotron-CC-Math, Specialized-v1.1 code and STEM mixes
Web search backend: Tavily
Inference hosting: Modal (L4 GPU)

License

MIT. Same as the nanochat base — see the project repository for the full text.

ManmohanSharma/samosachaat-d24