Model Card: Llama-3-Hercules-5.0-8B

Model Description

• Complex Instruction Following: Understanding and accurately executing multi-step instructions, even those involving specialized terminology.
• Function Calling: Seamlessly interpreting and executing function calls, providing appropriate input and output values.
• Domain-Specific Knowledge: Engaging in informative and educational conversations about Biology, Chemistry, Physics, Mathematics, Medicine, Computer Science, and more.

Intended Uses & Potential Bias

• Specialized Chatbots: Creating knowledgeable chatbots and conversational agents in scientific and technical fields.
• Instructional Assistants: Supporting users with educational and step-by-step guidance in various disciplines.
• Code Generation and Execution: Facilitating code execution through function calls, aiding in software development and prototyping.

Quants

Limitations and Risks

• Toxicity: The dataset contains toxic or harmful examples.
• Hallucinations and Factual Errors: Like other language models, Llama-3-Hercules-5.0-8B may generate incorrect or misleading information, especially in specialized domains where it lacks sufficient expertise.
• Potential for Misuse: The ability to engage in technical conversations and execute function calls could be misused for malicious purposes.

Training Procedure

• This model was trained on 8 kaggle TPUs, using torch xla SPMD for high MXU efficiency. There was no expense on my end (meaning you can reproduce this too!)
• A learning rate of 2e-5 with the Adam optimizer. A linear scheduler was used, with an end factor of 0.005.
• No mixed precision was used, with the default dtype being bfloat16.
• A total batch size of 128 was used.
• Trained on all examples of Hercules-v5.0 for 2 epochs
• No model parameters were frozen and no quantization was used.
• This model was trained on OpenAI's ChatML prompt format. Because this model has function calling capabilities, the prompt format is slightly different, here's what it would look like: <|im_start|>system\n{message}<|im_end|>\n<|im_start|>user\n{user message}<|im_end|>\n<|im_start|>call\n{function call message}<|im_end|>\n<|im_start|>function\n{function response message}<|im_end|>\n<|im_start|>assistant\n{assistant message}</s>