You can deploy this model and start earning money today!

small56ai/
ApproxDumb

Name: ApproxDumb
Rating: 2.5 (36 reviews)

Views

No views yet

ApproxDumb (560-Parameter Micro-SLM)

ApproxDumb is an ultra-micro language model built with just 560 parameters.
It serves as a proof-of-concept for In-Context Learning (ICL) in micro-architectures, demonstrating how a model can infer underlying rules (such as dynamic addition patterns) on-the-fly from a single provided example.

⚠️ Input & Output Constraints

Due to the extreme parameter budget, this model cannot process natural language text or complex multi-digit arithmetic. Please strictly adhere to the following input and output specifications.

1. Input Constraints

Allowed Vocabulary:
- Single Digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
- Separator: ->
- Special Tokens: <bos>, <eos>, <pad>
- Note: Words, letters, multi-digit numbers (e.g., 10, 100), or unsupported punctuation are NOT allowed.
Formatting:
- Tokens must be separated by half-width spaces (e.g., "1 3 -> 4").
- Recommended Prompt Structure: [Example Input] [Example Output] -> [Query Input] (Total of 4 tokens).
Max Sequence Length: 8 tokens (4 to 5 tokens recommended).

2. Output Constraints

Output Format: The model predicts the probability distribution (logits) for the next single digit token (0–9).
Limitations:
- Cannot generate multi-digit numbers ($\ge 10$) or negative numbers.
- Not designed for multi-token free-form text generation (specialized for single next-token prediction).

💡 Prompt Examples

By providing a single "Example", the model infers the rule and predicts the answer for the "Query".

Prompt (`input_text`)	Inferred Rule	Expected Prediction
`"1 2 -> 4"`	$+1$ rule	`5`
`"1 3 -> 4"`	$+2$ rule	`6`
`"1 4 -> 2"`	$+3$ rule	`5`
`"0 4 -> 1"`	$+4$ rule	`5`

🚀 How to Use

Make sure to pass trust_remote_code=True when loading the model and tokenizer.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model_id = "56m/ApproxDumb"  # Replace with your Hugging Face repository
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

# Prepare prompt: Example "1 -> 3" (+2 rule), Query "4 -> ?"
prompt = "1 3 -> 4"
inputs = tokenizer(prompt, return_tensors="pt")

# Inference
model.eval()
with torch.no_grad():
    outputs = model(**inputs)
    
# Extract the most probable next digit token from the final position
next_token_logits = outputs.logits[0, -1, :]
predicted_token_id = torch.argmax(next_token_logits).item()
predicted_symbol = tokenizer.decode([predicted_token_id])

print(f"Input: '{prompt}'")
print(f"Predicted Next Digit: {predicted_symbol}")

🏗️ Model Architecture

Architecture: Decoder-only Micro-Transformer
Total Parameters: 560
Embedding Dimension ($d_{model}$): 6
Transformer Layers: 1
Attention Heads: 1
Vocabulary Size: 14

small56ai/ApproxDumb