Addressing challenges such as inefficient knowledge organization and insufficient understanding of professional content in natural science academic text processing, this study constructed an academic large language model oriented towards the natural science domain—NSLLM
Developing based on Qwen3-8B-Base, the model utilizes a massive amount of natural science literature for a two-stage "Basic Adaptation + Domain Deepening" continue pre-training strategy. It aims to balance the retention of old knowledge with the acquisition of new knowledge, providing a feasible technical path for knowledge mining and research assistance in the natural science domain.
The study first constructed a pre-training corpus of academic texts covering nearly 50 years in the field of natural science, containing over 17 million papers. The data is primarily sourced from:
Web of Science
arXiv
PubMed
11个国际顶级学术会议 (11 Top International Academic Conferences)
核心能力
Core Capabilities
通过自然语言理解、自然语言生成及领域知识问答等维度的评测，NSLLM 展现出以下核心优势：
Through evaluations in dimensions such as natural language understanding, natural language generation, and domain knowledge Q&A, NSLLM demonstrates the following core advantages:
精准的领域适应性：在摘要学科分类、摘要语步识别等对领域知识精准性要求较高的任务中表现显著。
Precise Domain Adaptability: Performs significantly well in tasks requiring high precision in domain knowledge, such as abstract discipline classification and abstract move recognition.
高质量的学术生成：生成的摘要不仅能复现研究方法与结论，还能通过句式重构提升学术规范性。
High-Quality Academic Generation: The generated abstracts can not only reproduce research methods and conclusions but also enhance academic standardization through sentence restructuring.
3. 模型训练
3. Model Training
训练策略：两阶段继续预训练
Training Strategy: Two-Stage Continue Pre-training
为了解决从头训练的高成本及灾难性遗忘问题，本研究采用了两阶段训练策略：
To address the high costs of training from scratch and the issue of catastrophic forgetting, this study adopted a two-stage training strategy:
第一阶段：基础适应 (Phase 1: Basic Adaptation)
内容：利用自然科学领域标题与摘要语料。
Content: Utilizes titles and abstracts from the natural science domain.
设置：以初始学习率预训练 200 步。
Settings: Pre-trained for 200 steps with an initial learning rate.
目标：激活模型对科学术语的初步认知。
Goal: Activate the model's preliminary cognition of scientific terminology.
第二阶段：领域深化 (Phase 2: Domain Deepening)
内容：引入自然科学论文的主要实验部分内容。
Content: Introduces the main experimental sections of natural science papers.
目标：进一步强化模型对于自然科学领域知识的认知能力，优化模型输出的逻辑严谨性。
Goal: Further strengthen the model's cognitive ability regarding natural science domain knowledge and optimize the logical rigor of model outputs.
4. 模型效果
4. Model Performance
关键指标提升
Key Metrics Improvement
与同参数规模的 Qwen3-8B 基座模型相比，NSLLM 取得了显著提升：
Compared to the Qwen3-8B base model of the same parameter scale, NSLLM achieved significant improvements:
论文摘要生成 (Abstract Generation): BERTScore 达到 66.18%，较基座模型提升 0.59 个百分点 (BERTScore reached 66.18%, an increase of 0.59 percentage points over the base model).
下游任务对比
Downstream Task Comparison
我们在六大下游任务中进行了广泛的对比验证：
We conducted extensive comparative validation across six major downstream tasks:
Same Parameter Scale Comparison: NSLLM achieved the best or near-best performance in all tasks, outperforming general models like Mistral-7B-Instruct, Llama3.1-8B-Instruct, and GLM-4-9B.
Cross Parameter Scale Comparison: Thanks to pre-training on massive domain data, NSLLM demonstrates strong knowledge transfer capabilities, with performance approaching that of larger models in the same series (Qwen3-14B, Qwen3-32B).
NSLLM 模型下游任务验证对比 | Comparison of downstream task validation for NSLLM
This study verifies the effectiveness of domain-specific training in enhancing the professional capabilities of large language models. In the future, we will further explore interdisciplinary data fusion and knowledge update mechanisms to continuously improve the model's performance in research assistance scenarios.
The vast parameter size of large language models introduces significant randomness. While we have strived to ensure data compliance during training data selection, unavoidable issues may still exist due to the complexity of the data and the model. Therefore, we assume no responsibility for any issues arising from the use of this open-source model, including but not limited to data security issues, public opinion risks, or any risks and problems caused by the model being misled, misused, disseminated, or improperly utilized.
Furthermore, in accordance with the "Interim Measures for the Management of Generative Artificial Intelligence Services" jointly issued by the Cyberspace Administration of China and six other departments, please strictly adhere to relevant laws and regulations when training or using this model and other generative models. We call for joint efforts to build a harmonious, healthy, and sustainable generative AI community. Consequently, we assume no responsibility for any issues arising from the use of this open-source model, including but not limited to data security issues, public opinion risks, or any risks and problems caused by the model being misled, misused, disseminated, or improperly utilized.
