AstroLLaMA-3-8B-Chat_Summary is a specialized chat model for astronomy, developed by fine-tuning the AstroLLaMA-3-8B-Base_Summary model. This model was developed by the AstroMLab team. It is designed for instruction-following and chat-based interactions in the astronomy domain.
Model Details
Base Architecture: LLaMA-3-8b
Base Model: AstroLLaMA-3-8B-Base_Summary (trained on summarized content from arXiv's astro-ph category papers)
Data Processing:
Optical character recognition (OCR) on PDF files using the Nougat tool
Summarization of OCR'd text using Qwen-2-8B and LLaMA-3.1-8B, reducing content to about 1,000-4,000 tokens per paper
Fine-tuning Method: Supervised Fine-Tuning (SFT)
SFT Dataset:
10,356 astronomy-centered conversations generated from arXiv abstracts by GPT-4
Full content of LIMA dataset
10,000 samples from Open Orca dataset
10,000 samples from UltraChat dataset
Training Details:
Learning rate: 3 × 10⁻⁷
Training epochs: 1
Total batch size: 48
Maximum token length: 2048
Warmup ratio: 0.03
Cosine decay schedule for learning rate reduction
Primary Use: Instruction-following and chat-based interactions for astronomy-related queries
Reference: Pan et al. 2024 [Link to be added]
Using the model for chat
python
1from transformers import AutoModelForCausalLM, AutoTokenizer
2import torch
34# Load the model and tokenizer5tokenizer = AutoTokenizer.from_pretrained("AstroMLab/astrollama-3-8b-chat_summary")6model = AutoModelForCausalLM.from_pretrained("AstroMLab/astrollama-3-8b-chat_summary", device_map="auto")78# Function to generate a response9defgenerate_response(prompt, max_length=512):10 full_prompt =f"###Human: {prompt}\n\n###Assistant:"11 inputs = tokenizer(full_prompt, return_tensors="pt", truncation=True, max_length=max_length)12 inputs = inputs.to(model.device)1314# Generate a response15with torch.no_grad():16 outputs = model.generate(17**inputs,18 max_length=max_length,19 num_return_sequences=1,20 do_sample=True,21 pad_token_id=tokenizer.eos_token_id,22 eos_token_id=tokenizer.encode("###Human:", add_special_tokens=False)[0]23)2425# Decode and return the response26 response = tokenizer.decode(outputs[0], skip_special_tokens=False)2728# Extract only the Assistant's response29 assistant_response = response.split("###Assistant:")[-1].strip()30return assistant_response
3132# Example usage33user_input ="What are the main components of a galaxy?"34response = generate_response(user_input)35print(f"Human: {user_input}")36print(f"Assistant: {response}")
Model Improvements and Performance
This model used the summarized content for training, which has led to improved performance compared to the AIC (Abstract, Introduction, Conclusion) version. The summarization process allows for the inclusion of more comprehensive information from each paper while maintaining a manageable token count.
Here's a performance comparison chart based upon the astronomical benchmarking Q&A as described in Ting et al. 2024, and Pan et al. 2024:
Model
Score (%)
LLaMA-3.1-8B
73.7
LLaMA-3-8B
72.9
AstroLLaMA-3-8B-Base_Summary (AstroMLab)
72.3
AstroLLaMA-3-8B-Chat_Summary (AstroMLab)
69.0
Gemma-2-9B
71.5
Qwen-2.5-7B
70.4
Yi-1.5-9B
68.4
InternLM-2.5-7B
64.5
Mistral-7B-v0.3
63.9
ChatGLM3-6B
50.4
As shown, AstroLLaMA-3-8B-Chat_Summary performs competitively, maintaining most of the performance of the base summary model. This demonstrates the effectiveness of the summarization approach in capturing and retaining key astronomical concepts, even after fine-tuning for chat interactions.
We also found that the model trained with summaries leads to better scores in general, especially with the instruct version, demonstrating that information density matters significantly in specialized domain training.
While AstroLLaMA-3-8B-Chat_Summary performs well among models in its class, it does not surpass the performance of the base LLaMA-3.1-8B model. This underscores the ongoing challenges in developing specialized models and the need for continued research in this area.
This model is released primarily for reproducibility purposes, allowing researchers to track the development process and compare different iterations of AstroLLaMA models.
For optimal performance and the most up-to-date capabilities in astronomy-related tasks, we recommend using AstroSage-8B, where further improvements have been made. The newer model incorporates expanded training data beyond astro-ph and features a greatly expanded fine-tuning process, resulting in significantly improved performance.
Ethical Considerations
While this model is designed for scientific use, users should be mindful of potential misuse, such as generating misleading scientific content. Always verify model outputs against peer-reviewed sources for critical applications.
Citation
If you use this model in your research, please cite:
