This is a custom modification of the AiAsistent/Phi-4-mini-reasoning-heretic model created personally by AlexH. Please note that this customization is not part of the official release.

The update includes unique enhancements that improve flexibility, reduce refusals, and expand the model’s responsiveness in extreme or complex scenarios.

For a detailed explanation of how this update works, access to the code, and ongoing support, visit:
Heretic LLM Universal Support for New Models via Dynamic Auto-Registration

This custom version demonstrates the power of user-driven modifications to safely extend the capabilities of large language models. By applying advanced configuration tweaks and dynamic auto-registration, the model now provides a more robust and versatile experience while maintaining full compatibility with the Heretic ecosystem.

This is a decensored version of baidu/ERNIE-4.5-0.3B-PT, made using Heretic v1.1.0

Abliteration parameters

Parameter	Value
direction_index	per layer
attn.o_proj.max_weight	0.96
attn.o_proj.max_weight_position	13.90
attn.o_proj.min_weight	0.61
attn.o_proj.min_weight_distance	7.99
mlp.down_proj.max_weight	0.98
mlp.down_proj.max_weight_position	10.31
mlp.down_proj.min_weight	0.93
mlp.down_proj.min_weight_distance	9.94

Performance

Metric	This model	Original model (baidu/ERNIE-4.5-0.3B-PT)
KL divergence	0.6943	0 (by definition)
Refusals	27/100	62/100

Github

Discord

ERNIE-4.5-0.3B

[!NOTE] Note: "-Paddle" models use PaddlePaddle weights, while "-PT" models use Transformer-style PyTorch weights.

ERNIE 4.5 Highlights

The advanced capabilities of the ERNIE 4.5 models, particularly the MoE-based A47B and A3B series, are underpinned by several key technical innovations:

Multimodal Heterogeneous MoE Pre-Training: Our models are jointly trained on both textual and visual modalities to better capture the nuances of multimodal information and improve performance on tasks involving text understanding and generation, image understanding, and cross-modal reasoning. To achieve this without one modality hindering the learning of another, we designed a heterogeneous MoE structure, incorporated modality-isolated routing, and employed router orthogonal loss and multimodal token-balanced loss. These architectural choices ensure that both modalities are effectively represented, allowing for mutual reinforcement during training.
Scaling-Efficient Infrastructure: We propose a novel heterogeneous hybrid parallelism and hierarchical load balancing strategy for efficient training of ERNIE 4.5 models. By using intra-node expert parallelism, memory-efficient pipeline scheduling, FP8 mixed-precision training and finegrained recomputation methods, we achieve remarkable pre-training throughput. For inference, we propose multi-expert parallel collaboration method and convolutional code quantization algorithm to achieve 4-bit/2-bit lossless quantization. Furthermore, we introduce PD disaggregation with dynamic role switching for effective resource utilization to enhance inference performance for ERNIE 4.5 MoE models. Built on PaddlePaddle, ERNIE 4.5 delivers high-performance inference across a wide range of hardware platforms.
Modality-Specific Post-Training: To meet the diverse requirements of real-world applications, we fine-tuned variants of the pre-trained model for specific modalities. Our LLMs are optimized for general-purpose language understanding and generation. The VLMs focuses on visuallanguage understanding and supports both thinking and non-thinking modes. Each model employed a combination of Supervised Fine-tuning (SFT), Direct Preference Optimization (DPO) or a modified reinforcement learning method named Unified Preference Optimization (UPO) for post-training.

Model Overview

ERNIE-4.5-0.3B is a text dense Post-trained model. The following are the model configuration details:

Key	Value
Modality	Text
Training Stage	Posttraining
Params	0.36B
Layers	18
Heads(Q/KV)	16 / 2
Context Length	131072

Quickstart

Using `transformers` library

Note: You'll need the transformers library (version 4.54.0 or newer) installed to use this model.

The following contains a code snippet illustrating how to use the model generate content based on given inputs.

python

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "baidu/ERNIE-4.5-0.3B-PT"

# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# prepare the model input
prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], add_special_tokens=False, return_tensors="pt").to(model.device)

# conduct text completion
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=1024
)
output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()

# decode the generated ids
generate_text = tokenizer.decode(output_ids, skip_special_tokens=True)
print("generate_text:", generate_text)

vLLM inference

vllm github library. Python-only build.

vllm serve baidu/ERNIE-4.5-0.3B-PT

License

Citation

If you find ERNIE 4.5 useful or wish to use it in your projects, please kindly cite our technical report:

bibtex

@misc{ernie2025technicalreport,
      title={ERNIE 4.5 Technical Report},
      author={Baidu ERNIE Team},
      year={2025},
      eprint={},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={}
}

AiAsistent/ERNIE-4.5-0.3B-PT-heretic