Quin Beta 1.1
Quin Beta 1.1 is an advanced Transformer-based language model with 2.7 billion parameters. It is an extension of Quin Beta 1.1, trained on an enhanced dataset that includes synthetic NLP texts and curated web content. The model demonstrates near state-of-the-art performance in benchmarks for common sense, language understanding, and logical reasoning within its parameter class. Quin Beta 1.1 is designed for QA, chat, and code formats and has a context length of 2048 tokens.
- Key Features
- Parameter Count: 2.7 billion
- Training Data: 250 billion tokens from a combination of AOAI synthetic data and filtered web data
- Training Tokens: 1.4 trillion tokens
- Hardware Used: 96xA100-80G GPUs
- Training Duration: 14 days
- Frameworks: ONNX and JAX
- Reinforcement Learning: Not applied (intended for enabling safety research)
- Applications: QA, chat, and code formats
- Context Length: 2048 tokens
- Open-Source: Yes, aimed at enabling safety research like toxicity reduction and bias understanding
Performance
Quin Beta 1.1 has demonstrated near state-of-the-art performance on various benchmarks, specifically targeting:
- Common sense reasoning
- Language understanding
- Logical reasoning
Usage
Installation
To use Quin Beta 1.1, you will need to have ONNX and JAX installed. Follow the installation instructions for each framework:
ONNX
JAX
Loading the Model
Here is a basic example of how to load and use Quin Beta 1.1 using ONNX:
1import onnxruntime as ort
2
3# Load the ONNX model
4ort_session = ort.InferenceSession("quin_beta_1.1.onnx")
5
6# Prepare input
7input_text = "Your input text here."
8inputs = {
9 "input_ids": [input_text],
10}
11
12# Run inference
13outputs = ort_session.run(None, inputs)
14
15# Get the result
16result = outputs[0]
17print(result)
18
Research and Safety
Quin Beta 1.1 is open-source and designed to enable research in safety domains such as:
- Toxicity reduction
- Bias understanding and mitigation
The model has not undergone reinforcement learning fine-tuning, providing a baseline for safety research efforts.
Contributing
Contributions to Quin Beta 1.1 are welcome. If you would like to contribute, please follow these steps:
- Fork the repository.
- Create a new branch (git checkout -b feature-branch).
- Make your changes.
- Commit your changes (git commit -am 'Add new feature').
- Push to the branch (git push origin feature-branch).
- Create a new Pull Request.
License
Quin Beta 1.1 is released under the apache-2.0 License. See the LICENSE file for more details.
Contact
For any questions or issues, please contact
drchamyoung@gmail.com
