Made with Exllamav2 0.2.8 with default calibration dataset.
Exl2 models can be used with TabbyAPI, Text-Generation-WebUI and some other apps.
It requires to use Nvidia RTX GPU on Windows or Nvidia RTX/AMD ROCm on Linux.
Model has to fit VRAM to be used properly. If it's bigger than your GPU can handle, please use GGUF quants for llama.cpp apps instead.

Original model card

Galactic-Qwen-14B-Exp2

Galactic-Qwen-14B-Exp2 is based on the Qwen 2.5 14B modality architecture, designed to enhance the reasoning capabilities of 14B-parameter models. This model is optimized for general-purpose reasoning and answering, excelling in contextual understanding, logical deduction, and multi-step problem-solving. It has been fine-tuned using a long chain-of-thought reasoning model and specialized datasets to improve comprehension, structured responses, and conversational intelligence.

Key Improvements

Enhanced General Knowledge: The model provides broad knowledge across various domains, improving capabilities in answering questions accurately and generating coherent responses.
Improved Instruction Following: Significant advancements in understanding and following complex instructions, generating structured responses, and maintaining coherence over extended interactions.
Versatile Adaptability: More resilient to diverse prompts, enhancing its ability to handle a wide range of topics and conversation styles, including open-ended and structured inquiries.
Long-Context Support: Supports up to 128K tokens for input context and can generate up to 8K tokens in a single output, making it ideal for detailed responses.
Multilingual Proficiency: Supports over 29 languages, including English, Chinese, French, Spanish, Portuguese, German, Italian, Russian, Japanese, Korean, Vietnamese, Thai, Arabic, and more.

Quickstart with transformers

Here is a code snippet with apply_chat_template to show you how to load the tokenizer and model and generate content:

python

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "prithivMLmods/Galactic-Qwen-14B-Exp2"

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "What are the key principles of general-purpose AI?"
messages = [
    {"role": "system", "content": "You are a helpful assistant capable of answering a wide range of questions."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

Intended Use

General-Purpose Reasoning:
Designed for broad applicability, assisting with logical reasoning, answering diverse questions, and solving general knowledge problems.
Educational and Informational Assistance:
Suitable for providing explanations, summaries, and research-based responses for students, educators, and general users.
Conversational AI and Chatbots:
Ideal for building intelligent conversational agents that require contextual understanding and dynamic response generation.
Multilingual Applications:
Supports global communication, translations, and multilingual content generation.
Structured Data Processing:
Capable of analyzing and generating structured outputs, such as tables and JSON, useful for data science and automation.
Long-Form Content Generation:
Can generate extended responses, including articles, reports, and guides, maintaining coherence over large text outputs.

Limitations

Hardware Requirements:
Requires high-memory GPUs or TPUs due to its large parameter size and long-context support.
Potential Bias in Responses:
While designed to be neutral, outputs may still reflect biases present in training data.
Inconsistent Outputs in Creative Tasks:
May produce variable results in storytelling and highly subjective topics.
Limited Real-World Awareness:
Does not have access to real-time events beyond its training cutoff.
Error Propagation in Extended Outputs:
Minor errors in early responses may affect overall coherence in long-form outputs.
Prompt Sensitivity:
The effectiveness of responses may depend on how well the input prompt is structured.

Open LLM Leaderboard Evaluation Results

Detailed results can be found here! Summarized results can be found here!

Metric	Value (%)
Average	43.56
IFEval (0-Shot)	66.20
BBH (3-Shot)	59.92
MATH Lvl 5 (4-Shot)	34.74
GPQA (0-shot)	19.91
MuSR (0-shot)	28.49
MMLU-PRO (5-shot)	52.12

cgus/Galactic-Qwen-14B-Exp2-exl2

Galactic-Qwen-14B-Exp2-exl2

Quants

Quantization notes