Beta
Explore
Marketplace
Neural Labs
Chat
Wallet
Docs
gpt-neoxt-20b-v12 – AI Model by iamplus | AlphaNeural AI
You can deploy this model and start earning money today!
iamplus
/
gpt-neoxt-20b-v12
like
0
transformers
pytorch
gpt_neox
text-generation
iamplus/Instruction_Tuning
iamplus/Conversational_Data
bigscience-openrail-m
autotrain_compatible
text-generation-inference
endpoints_compatible
us
Views
No views yet
Model card
Files and Versions
Community
API
Deploy
Instruction Tuned GPT-NeoXT-20B model on Instruction Tuning dataset as listed below (~5.2M data) using
Colossal AI
Base Model:
togethercomputer/GPT-NeoXT-Chat-Base-20B (GPT-NeoXT-Chat-Base-20B-v0.16 - fine-tuned on feedback data)
Training Details :
Epochs: 4
Batch Size : 5 instantaneous per device x 1 gradient accumulation steps x 8 gpus = 40
Block Size : 2020
Weight Decay : 0
Learning Rate : 1e-6
Learning Rate Scheduler Type : Cosine
Number of warmup steps : 600
Machine : 8xA100 80GB
Training Data Specifics :
Labels and Input ids are exactly the same.
Block Size is 2020, Multiple instructions are clubbed together in each data.
"###" is the EOS Token used in the data.
