hou-rai (蓬莱) a150m 7.8b Trainig codes | Other

Model Details

file tree

# ==================================================
./scripts
  ./merge_150m_step1_distill.py
  ./merge_150m_step2_wikipedia.py
  ./merge_150m_step3_aozora.py
  ./merge_150m_step3_aozora_fix.py

  ./train_peft_150m_step1_distll.py
  ./train_peft_150m_step2_wikipedia.py
  ./train_peft_150m_step3_aozora.py
  ./train_peft_150m_step3_aozora_fix.py

# ==================================================
# datasets
# ==================================================
./datasets
  ./Japanese-Wikipedia-202506
    ./
  ./aozorabunko-clean/aozorabunko-dedupe-clean.jsonl.gz
  ./Zero_SFT_Ja_v3.5_Reasoning/answers_bare.jsonl
  ./Synthetic-JP-EN-Coding-Dataset-801k/train-00000-of-00013.parquet
  ./wizardlm8x22b-logical-math-coding-sft-ja/train-00000-of-00002.parquet

./mergekit/
  ./config.yml
  ./split1.yml

# ==================================================
# model weights
# ==================================================

./qwen3-150x52m-7.8b
  ./chat_template.jinja
  ./config.json
  ./mergekit_moe_config.yml
  ./model.safetensors.index.json
  ./model-00001-of-00002.safetensors
  ./model-00002-of-00002.safetensors
  ./tokenizer.json
  ./tokenizer_config.json

./qwen3-150x52m-7.8b-step1-distll
  ./chat_template.jinja
  ./config.json
  ./generation_config.json
  ./model.safetensors
  ./tokenizer.json
  ./tokenizer_config.json

./qwen3-150x52m-7.8b-step2-wikipedia
  ./chat_template.jinja
  ./config.json
  ./generation_config.json
  ./model.safetensors
  ./tokenizer.json
  ./tokenizer_config.json

./qwen3-150x52m-7.8b-step3-aozora
  ./chat_template.jinja
  ./config.json
  ./generation_config.json
  ./model.safetensors
  ./tokenizer.json
  ./tokenizer_config.json

./qwen3-150x52m-7.8b-step3-aozora-fix
  ./chat_template.jinja
  ./config.json
  ./generation_config.json
  ./model.safetensors
  ./tokenizer.json
  ./tokenizer_config.json

# ==================================================
# model gguf (can work upstream llama.cpp)
# ==================================================

Model Description

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

Direct Use

Downstream Use [optional]

Out-of-Scope Use

Bias, Risks, and Limitations

Recommendations

How to Get Started with the Model

Training Details

Training Data

• OmniAICreator/Japanese-Wikipedia-202506
• globis-university/aozorabunko-clean

• DataPilot/Zero_SFT_Ja_v3.5_Reasoning

• llm-jp/wizardlm8x22b-logical-math-coding-sft-ja
• Aratako/Synthetic-JP-EN-Coding-Dataset-801k

Training Procedure

Preprocessing [optional]

Training Hyperparameters

• Training regime: [More Information Needed]
• qwen3-150x52m-step1-distill
  model (30000steps checkpoints)

  dataset

  • DataPilot/Zero_SFT_Ja_v3.5_Reasoning

  #### lora config
  
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "gate_up_proj"
    ]
    lora_dropout=0,
    bias="none",
  
  #### transformer trainer config
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4, 
    warmup_steps=5,
    max_steps=50000,
    learning_rate=2e-4,
    optim="adamw_8bit",             
    weight_decay=0.001,
    lr_scheduler_type="linear",
    seed=3407,
    fp16=False,                     
    bf16=True,                     
    remove_unused_columns=False

• qwen3-150x52m-step2-wikipedia
  model (10000steps checkpoints)

  dataset

  • DataPilot/Zero_SFT_Ja_v3.5_Reasoning [:30000]
  • OmniAICreator/Japanese-Wikipedia-202506 [:30000]

  #### lora config
  
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "gate_up_proj"
    ]
    lora_dropout=0,
    bias="none",
  
  #### transformer trainer config
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4, 
    warmup_steps=5,
    max_steps=50000,
    learning_rate=2e-4,
    optim="adamw_8bit",             
    weight_decay=0.001,
    lr_scheduler_type="linear",
    seed=3407,
    fp16=False,                     
    bf16=True,                     
    remove_unused_columns=False

• qwen3-150x52m-step3-aozora
  model (15000steps checkpoints)

  dataset

  • DataPilot/Zero_SFT_Ja_v3.5_Reasoning [:30000]
  • OmniAICreator/Japanese-Wikipedia-202506 [:30000]
  • globis-university/aozorabunko-clean [:15000]

  #### lora config
  
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "gate_up_proj"
    ]
    lora_dropout=0,
    bias="none",
  
  #### transformer trainer config
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4, 
    warmup_steps=5,
    max_steps=50000,
    learning_rate=2e-4,
    optim="adamw_8bit",             
    weight_decay=0.001,
    lr_scheduler_type="linear",
    seed=3407,
    fp16=False,                     
    bf16=True,                     
    remove_unused_columns=False

• qwen3-150x52m-step3-aozora-fix
  model (45000steps checkpoints)

  dataset

  • DataPilot/Zero_SFT_Ja_v3.5_Reasoning [:30000]
  • OmniAICreator/Japanese-Wikipedia-202506 [:30000]
  • globis-university/aozorabunko-clean [:15000]

  #### lora config
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "gate_up_proj"
    ]
    lora_dropout=0,
    bias="none",
  
  #### SFTtrainer SFTConfig
        max_length=max_8192,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=1,
        warmup_steps=0,
        max_steps=60000,
        learning_rate=5e-4,
        logging_steps=5,
        optim="adamw_8bit",            
        weight_decay=0.0005,
        lr_scheduler_type="linear",
        seed=3407,
        report_to="none",
        bf16=True,                    
        remove_unused_columns=False,
        completion_only_loss=True

• qwen3-150x52m-step4_math

  dataset

  • DataPilot/Zero_SFT_Ja_v3.5_Reasoning [:10000]
  • OmniAICreator/Japanese-Wikipedia-202506 [:10000]
  • globis-university/aozorabunko-clean [:10000]
  • llm-jp/wizardlm8x22b-logical-math-coding-sft-ja [:10000]
  • Aratako/Synthetic-JP-EN-Coding-Dataset-801k [:10000]

  #### lora config
    task_type=TaskType.CAUSAL_LM,
    r=16,
    lora_alpha=32,
    target_modules=[
        "q_proj",
        "k_proj",
        "v_proj",
        "o_proj",
        "gate_proj",
        "up_proj",
        "down_proj",
        "gate_up_proj"
    ]
    lora_dropout=0,
    bias="none",
  
  #### SFTtrainer SFTConfig
        max_length=8192,
        per_device_train_batch_size=1,
        gradient_accumulation_steps=1,
        warmup_steps=0,
        max_steps=60000,
        learning_rate=5e-4,
        logging_steps=5,
        optim="adamw_8bit",            
        weight_decay=0.0005,
        lr_scheduler_type="linear",
        seed=3407,
        report_to="none",
        bf16=True,                    
        remove_unused_columns=False,
        completion_only_loss=True

• qwen3-150x52m-step4-math-fix

Speeds, Sizes, Times [optional]

Evaluation

Testing Data, Factors & Metrics

Testing Data

Factors

Metrics

Results

Summary

Model Examination [optional]

Environmental Impact

• Hardware Type: [More Information Needed]
• Hours used: [More Information Needed]
• Cloud Provider: [More Information Needed]
• Compute Region: [More Information Needed]
• Carbon Emitted: [More Information Needed]

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

hou-rai (蓬莱) 2 Trainig codes | Other

Technical Specifications [optional]

Model Architecture and Objective

Compute Infrastructure

Hardware

Software

Citation [optional]

Glossary [optional]

More Information [optional]

Model Card Authors [optional]

Model Card Contact