Important!: Currently, this model has not undergone CPT (Continued Pre-Training) following the merge; it will be replaced later with a CPT-processed version.

PLaMo 3 12B Self-Merged Base is a model created by self-merging pfnet/plamo-3-nict-8b-base and scaling it up to 12B parameters.

PLaMo 3 12B Self-Merged Base currently has some unstable elements, which we plan to address in an upcoming update.

PLaMo 3 12B Self-Merged Base is released under the PLaMo Community License. Please review the license below and agree to its terms before downloading.

NOTE: This model has NOT been instruction-tuned for chat dialogue or other downstream tasks.

This is a merge of pre-trained language models created using mergekit.

For commercial use

Please review the PLaMo Community License and contact us using the form below if you intend to use this model for commercial purposes.

(EN/JA) https://forms.gle/mTL8tBLrMYXKNZD56

Usage

Requirements

python>=3.10.0
numpy>=1.26.4
numba>=0.60.0
torch>=2.6.0,<=2.9.0
transformers>=4.56.0,<=4.57.1

Loading the model directly

python

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "midorin-Linux/plamo-3-12b-self-merged"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

text = "これからの人工知能技術は"
inputs = tokenizer(text, return_tensors="pt").to(model.device)

generated_tokens = model.generate(
    **inputs,
    max_new_tokens=32,
    do_sample=True,
    top_k=50,
    top_p=0.95,
    temperature=1.0,
    pad_token_id=tokenizer.eos_token_id,
    use_cache=True,
)[0]

generated_text = tokenizer.decode(generated_tokens, skip_special_tokens=True)
print(generated_text)

Model Details

Model size: 12B (8B + 8B/2)
Base model (pfnet/plamo-3-nict-8b-base) developed by:
- Preferred Networks, Inc.
- In collaboration with the National Institute of Information and Communications Technology (NICT)
Model type: Causal decoder-only
Language(s): English, Japanese
License: PLaMo Community License

Base Model Information

For details on the base model, please refer to pfnet/plamo-3-nict-8b-base.

Built with PLaMo

midorin-Linux/plamo-3-12b-self-merged-base

PLaMo 3 12B Self-Merged Base