HRM Tiny JA 500k 2x2 RC1 / 非公式 HRM-Text 小型日本語 RC1
Important / 重要 unofficial experimental model . It is not released, endorsed, or certified by Sapient Intelligence, Hugging Face, or LLM-jp.非公式の実験モデル です。Sapient Intelligence、Hugging Face、LLM-jp による公式モデル、公式派生、認定モデルではありません。
This repository draft is for the release candidate selected from:
./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2/checkpoint-62500Suggested public-facing name:
This checkpoint is intended as a small Japanese instruction / explanation generation experiment based on the HF HRM-Text implementation. It is not a general-purpose chat model and should not be used as a factual authority.
1. Model summary / モデル概要
Item Value Architecture HRM-Text style causal LM, PrefixLM prompt masking Size class Tiny experimental model, approximately 120M parameter class Hidden size 512 Vocab size 99,584 model vocab rows Tokenizer length 99,574 effective tokenizer length Tokenizer llm-jp/llm-jp-3-150m-instruct3 tokenizerInput embeddings initialized from llm-jp/llm-jp-3-150m-instruct3 input embeddings LM head untied, randomly initialized before training Layers num_layers_per_stack=2, L stack 2 real layers, H stack 2 real layersH/L cycles H_cycles=2, L_cycles=2PrefixLM enabled; prompt tokens use token_type_ids=1, response tokens use token_type_ids=0 Final selected checkpoint checkpoint-62500, corresponding to roughly 1 epoch over 500k examples with batch size 8
日本語説明生成の検証用途を主目的とした小型モデルです。RC1では、2 epoch 学習の最終checkpointではなく、1 epoch相当の checkpoint-62500 を採用しています。1.5 epoch候補はfirst-token指標がやや良い一方で、本文がプロンプトから飛びやすかったため採用していません。
2. Prompt format / プロンプト形式
Training and evaluation used the following plain instruction format.
1 <s>以下は、タスクを説明する指示です。要求を適切に満たす応答を書きなさい。
2
3 ### 指示:
4 {instruction}
5
6 ### 応答:
The model was trained with PrefixLM-style masks. In local inference, prompt tokens should be assigned token_type_ids=1, and generated response tokens should be assigned token_type_ids=0.
3. Quick start / 推論サンプル
Install
pip install -U "git+https://github.com/huggingface/transformers.git" torch safetensorsHRM-Text support may require a recent Transformers version. The official HRM-Text documentation shows the general pattern of using AutoTokenizer and AutoModelForCausalLM.generate(...) for HRM-Text style models, while noting that the official HRM-Text-1B base model does not ship a stable chat template. For this RC, use the plain prompt above and the bundled inference script if PrefixLM token_type_ids handling is needed.
Recommended local script
1 python scripts/inference_hrm_prefixlm.py \
2 --model_dir ./release/hrm_tiny_ja_500k_2x2_rc1 \
3 --instruction "自然言語処理とは何か" \
4 --max_new_tokens 128 \
5 --temperature 0.65 \
6 --top_p 0.9 \
7 --top_k 50 \
8 --repetition_penalty 1.2 \
9 --no_repeat_ngram_size 4
Python API example
1 from pathlib import Path
2 from scripts . inference_hrm_prefixlm import load_model_and_tokenizer , generate_text
3
4 model_dir = Path ( "./release/hrm_tiny_ja_500k_2x2_rc1" )
5 model , tokenizer , device = load_model_and_tokenizer ( model_dir )
6
7 text = generate_text (
8 model = model ,
9 tokenizer = tokenizer ,
10 device = device ,
11 instruction = "人工知能と機械学習の違いを説明してください" ,
12 max_new_tokens = 128 ,
13 temperature = 0.65 ,
14 top_p = 0.9 ,
15 top_k = 50 ,
16 repetition_penalty = 1.2 ,
17 no_repeat_ngram_size = 4 ,
18 )
19 print ( text )
4. Training data / 学習データ
The RC1 dataset is:
./data/hrm_text_llmjp_tok_1024_release_ja_500k_cleanTarget composition after filtering and source sampling:
Validation splits:
Split Size / target Notes validation3,500 source-balanced clean validation validation_core500 fixed sanity prompts plus clean validation samples validation_magpie700 source-specific validation validation_aya700 source-specific validation validation_oasst2600 source-specific validation validation_oasst1500 source-specific validation validation_dolly_clean500 cleaned Dolly validation alias
Key filters:
Require Japanese text in instruction and response.
Drop numeric-only, symbol-heavy, refusal/meta, and apology-style responses.
Drop many list-like or numbered responses.
Drop repeated phrase / repeated n-gram responses.
Drop long context extraction tasks and prompt-heavy examples.
Token constraints: default min_response_tokens=24, max_response_tokens=320, max_prompt_tokens=448, max_total_tokens=896.
Banned first response tokens include Output, Solution, first, second, はい, いいえ, 以下, list markers, and numeric starts 0 through 20.
See cards/DATASET_CARD_DRAFT.md for a fuller dataset description.
5. Training setup / 学習設定
Selected RC1 run:
1 CUDA_VISIBLE_DEVICES = 0 accelerate launch --num_processes 1 train_hrm_tiny_sft.py \
2 --dataset_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean \
3 --eval_split_name validation_core \
4 --output_dir ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2 \
5 --num_layers_per_stack 2 \
6 --max_epochs 2 \
7 --per_device_train_batch_size 8 \
8 --gradient_accumulation_steps 1 \
9 --learning_rate 2e-4 \
10 --weight_decay 0.05 \
11 --warmup_ratio 0.03 \
12 --loss_normalization active_tokens \
13 --first_response_token_weight 4 \
14 --first_response_token_target first_non_space \
15 --embedding_init_mode raw \
16 --lm_head_init_mode random \
17 --eval_strategy epoch \
18 --save_strategy epoch \
19 --save_total_limit 4
Training metrics for the full 2-epoch run:
Metric Value epoch 2.0 train_loss 1.1787 train_runtime 1:48:06.09 train_samples_per_second 154.176 train_steps_per_second 19.272
Selected checkpoint:
6. Evaluation summary / 評価要約
Primary evaluation on release_ja_500k_clean / validation_core, first 256 examples:
Metric Value mean loss 3.322312 perplexity 27.724 active label tokens 32,079 active tokens / example 125.309 first-token mean loss 4.430201 first-token perplexity 83.948 first-token top1 / top5 / top20 45.703% / 57.422% / 69.922%
Legacy comparison on old mixfix_20k_ja4x / validation_ja_dolly, first 256 examples:
Metric Value mean loss 5.383554 perplexity 217.795 active label tokens 22,463 active tokens / example 87.746
Known behavior:
The model often selects a relevant first token for definition/explanation prompts.
It can produce Japanese explanation-like text.
It is still weak at factual accuracy, multi-sentence coherence, and causal explanations.
It can hallucinate details, confuse related concepts, or drift into generic explanations.
It is not suitable for safety-critical or factual-critical use.
See cards/EVAL_SUMMARY.md for selected logs and comparison notes.
To prepate this dataset, you can use prepare_release_ja_dataset_500k.py.
python prepare_release_ja_dataset_500k.py \
--output_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean \
--num_proc 8 \
--overwriteより厳しめに長文contextや長い応答を落とす場合はこちらです。
python prepare_release_ja_dataset_500k.py \
--output_dir ./data/hrm_text_llmjp_tok_1024_release_ja_500k_clean_strict \
--max_context_chars 700 \
--max_total_tokens 768 \
--max_response_tokens 256 \
--num_proc 8 \
--overwrite7. Example generations / 生成例
Generation parameters used for sanity checks:
1 temperature=0.65
2 top_p=0.9
3 top_k=50
4 repetition_penalty=1.15 or 1.2
5 no_repeat_ngram_size=3 or 4
6 max_new_tokens=128
Example prompt:
Observed RC1-style output excerpt:
自然言語処理（NLP）は、人間が言葉を理解し、生成するための技術で、...The sample is not guaranteed to be factually accurate. Use the model as an experimental small HRM-Text checkpoint, not as a factual assistant.
8. Limitations / 制限事項
非公式・研究用の小型モデルです。
事実誤認、幻覚、定義の混線、文章の途中崩れがあります。
日本語説明生成に寄せていますが、チャットモデルとして安定していません。
長文コンテキスト、厳密な数学、コード生成、翻訳、要約、抽出には向いていません。
医療・法律・金融・安全に関わる判断には使わないでください。
Dataset and upstream license compatibility must be checked before public redistribution.
9. Release checklist / 公開前チェック
Before public upload:
Copy checkpoint-62500 into a clean release directory.
Copy tokenizer files from the training output directory.
Run scripts/evaluate_rc1.sh and save logs under eval_logs/.
Confirm source dataset licenses and redistribution constraints.
Update model repo ID, author, date, and license field.
Add clear disclaimer: unofficial, experimental, no warranty.
Suggested packaging command:
1 bash scripts/package_rc1_checkpoint.sh \
2 ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2/checkpoint-62500 \
3 ./outputs/hrm_tiny_release_ja_500k_clean_2x2_semftw4_ep2 \
4 ./release/hrm_tiny_ja_500k_2x2_rc1
