Analysis of Quantization Impact on GPT-2

Results

Without Quantization

• Time Taken: 23.36 seconds
• Memory Usage: 510.34 MB
• Perplexity Score: 25.1880

With 8-Bit Integer Quantization

• Time Taken: 24.81 seconds
• Memory Usage: 255.87 MB
• Perplexity Score: 25.2121

With 8-Bit Quantization

• Time Taken: 30.73 seconds
• Memory Usage: 176.53 MB
• Perplexity Score: 25.3438

With 4-Bit Quantization (Without NF4)

• Time Taken: 23.61 seconds
• Memory Usage: 134.06 MB
• Perplexity Score: 29.7969

With 4-Bit Quantization (With NF4)

• Time Taken: 23.60 seconds
• Memory Usage: 134.06 MB
• Perplexity Score: 27.1406

1. Time Taken

• Baseline (No Quantization):
  Fastest time at 23.36 seconds, since no additional computations are required to handle quantized weights.
• 8-Bit Integer Quantization:
  Slightly slower at 24.81 seconds, likely due to overhead from managing integer operations.
• 8-Bit Quantization:
  30.73 seconds, showing a significant increase. This could be due to the GPU optimization overhead for mixed-precision calculations.
• 4-Bit Quantization (With and Without NF4):
  Both performed comparably around 23.60 seconds, indicating that further reduction in precision does not impact runtime significantly compared to the baseline.

2. Perplexity Score

• Baseline (No Quantization):
  Achieved the best perplexity score of 25.1880, reflecting full precision's benefit in maintaining model accuracy.
• 8-Bit Integer Quantization:
  Slightly worse at 25.2121, a negligible difference indicating that the model's accuracy is largely preserved.
• 8-Bit Quantization:
  Achieved a slightly higher perplexity of 25.3438, showing minor accuracy degradation due to the reduced precision.
• 4-Bit Quantization (Without NF4):
  Perplexity increased significantly to 29.7969, reflecting a notable drop in model performance.
• 4-Bit Quantization (With NF4):
  Perplexity of 27.1406, better than without NF4, indicating NF4's ability to mitigate some performance degradation.

3. Memory Usage

• Baseline (No Quantization):
  Highest memory usage at 510.34 MB, representing the full precision model size.
• 8-Bit Integer Quantization:
  Reduced memory by 50% to 255.87 MB, demonstrating the expected memory saving of integer quantization.
• 8-Bit Quantization:
  Further reduced memory to 176.53 MB, showing more efficient memory usage for floating-point quantization.
• 4-Bit Quantization (With and Without NF4):
  Achieved the lowest memory usage of 134.06 MB, representing a 73% reduction compared to the baseline.

Conclusion

• Time Efficiency:
  The 4-bit quantization (both with and without NF4) and baseline model are the fastest, while 8-bit quantization introduces overhead.
• Model Accuracy:
  The baseline and 8-bit integer quantization maintain high accuracy, while 4-bit quantization significantly impacts perplexity. Using NF4 improves the performance of 4-bit quantization.
• Memory Usage:
  As expected, lower-bit quantization reduces memory usage, with 4-bit quantization achieving the most significant savings.

Answer to some questions

Explain the concept of NF4 quantization and how it differs from linear quantization scales.

NF4 Quantization vs Linear Quantization

Discuss the impact of linear vs. nonlinear quantization on model accuracy and efficiency.

• 4-Bit Quantization (Without NF4):
  Perplexity increased significantly to 29.7969, reflecting a notable drop in model performance.
• 4-Bit Quantization (With NF4):
  Perplexity of 27.1406, better than without NF4, indicating NF4's ability to mitigate some performance degradation.