Report: Generated Image Quality Assessment Model
1. Introduction
This report details the development of a model for evaluating the quality of AI-generated human images. The evaluation criteria include:
- Relevance to the text prompt
- Visual quality and aesthetics
- Presence of artifacts and distortions
- Overall Quality
The workflow consists of three key stages:
- Dataset Preparation using ChatGPT VLM and CLIP embeddings.
- Model Training of distilled model leveraging CLIP features.
- Evaluation of model performance.
Due to cost and resource constraints, only 10,000 images were sourced from DiffusionDB, filtered to retain only human-related images. After removing invalid or problematic images, the final dataset size was 9,432.
Data Processing Workflow
- Filtering Prompts: Extracting images with human-related keywords (e.g., "man", "woman", "child").
- Downloading Images: Retrieving images from DiffusionDB based on filtered metadata.
- Generating Annotations Using ChatGPT VLM:
- GPT-4o Mini was used to assess:
- Relevance to the prompt (0.0 - 1.0 scale)
- Visual quality (composition, clarity, aesthetics)
- Presence of artifacts (missing parts, distortions, unnatural anatomy). Important note: 1 indicates artifact presence, 0 indicates absence. This approach was chosen as detecting artifact presence is significantly easier for the network compared to detecting their absence.
- Generating CLIP Embeddings:
- Image and text embeddings were extracted using CLIP ViT-B/32.
- Dataset Splitting:
- 80% training, 15% validation, and 5% test split.
Dataset Analysis
A histogram-based analysis was performed to examine score distributions. The dataset exhibited high imbalance, with some quality scores appearing far more frequently than others. To address this, weighted binary cross-entropy was employed during training.
The final dataset used in this study can be accessed at:
- Dataset (data.pkl): See here
A lightweight regression model was trained using CLIP embeddings to predict quality scores for new images.
Model Architecture
- Input: Concatenated CLIP image and text embeddings.
- Output: Predictions for four quality metrics.
Network Structure:
1Linear(1024, 256) → ReLU
2Linear(256, 4) → Sigmoid
Training Setup
- Optimizer: Adam (learning rate = 1e-4)
- Loss Function: Weighted Binary Cross-Entropy (compensating for class imbalance)
- Epochs: 100 (early stopping applied)
- Batch Size: 512
The trained model was tested on a held-out test dataset, and its performance was analyzed across multiple metrics.
Model Performance Metrics
The evaluation was conducted using multiple metrics to assess accuracy, precision, recall, and F1-score across different categories.
Training Results
| Metric | Overall |
|---|
| Accuracy | 0.8326 |
| Precision | 0.9930 |
| Recall | 0.9576 |
| F1 Score | 0.9746 |
Per-Category Metrics:
| Category | Accuracy | Precision | Recall | F1 Score |
|---|
| Relevance | 0.9633 | 0.9887 | 0.9712 | 0.9799 |
| Visual Quality | 0.9895 | 0.9990 | 0.9903 | 0.9947 |
| Artifacts | 0.8860 | 0.9895 | 0.8881 | 0.9360 |
| Final Probability | 0.9765 | 0.9948 | 0.9808 | 0.9878 |
Validation Results
| Metric | Overall |
|---|
| Accuracy | 0.7603 |
| Precision | 0.9699 |
| Recall | 0.9466 |
| F1 Score | 0.9579 |
Per-Category Metrics:
| Category | Accuracy | Precision | Recall | F1 Score |
|---|
| Relevance | 0.9151 | 0.9553 | 0.9508 | 0.9530 |
| Visual Quality | 0.9682 | 0.9883 | 0.9790 | 0.9836 |
| Artifacts | 0.8571 | 0.9607 | 0.8840 | 0.9208 |
| Final Probability | 0.9505 | 0.9755 | 0.9726 | 0.9740 |
Test Results
Overall Metrics
| Metric | Value |
|---|
| Accuracy | 0.7611 |
| Precision | 0.9741 |
| Recall | 0.9480 |
| F1 Score | 0.9606 |
Per-Category Metrics
| Category | Accuracy | Precision | Recall | F1 Score |
|---|
| Relevance | 0.9281 | 0.9652 | 0.9563 | 0.9607 |
| Visual Quality | 0.9767 | 0.9913 | 0.9848 | 0.9881 |
| Artifacts | 0.8478 | 0.9577 | 0.8750 | 0.9145 |
| Final Probability | 0.9598 | 0.9822 | 0.9757 | 0.9790 |
The model corresponding to the reported results can be accessed at:
- Best Model (model.pth): See here
5. Conclusion
The model achieved promising results given the imposed constraints on cost, time and computational resources. Despite these limitations, the evaluation metrics indicate that the approach is effective for assessing AI-generated human images.
Constraints and Future Improvements
- GPT-4o Mini was chosen over GPT-4o due to its significantly lower cost and faster processing speed. However, GPT-4o is a much more powerful model and could improve results significantly if used.
- The dataset size was capped at 10,000 images due to computational and financial constraints. A larger dataset would likely enhance performance.
- A lightweight head network was selected to balance computational feasibility with effective learning. A more complex model could further improve results but requires more computational power.
By addressing these areas, the model can achieve significantly better results in real-world applications.
