The dataset was preprocessed with the following steps:
Extract and structure the conversation data using role-based templates (SYSTEM, USER, ASSISTANT).
Convert the structured conversations into a tokenized format using the model's tokenizer.
Filter out sequences shorter than 4096 tokens.
Shuffle and select 512 samples for calibration.
Quantization Process
View the shell and python script used to quantize this model.
4 A40s with 300gb of ram was rented on runpod.
Quantization took approximately 11 hours with a total of $23.65 in compute costs. (And another $70 of me screwing up the quants like 10 times but anyways...)
