i3-CLIP: Hybrid RWKV-Attention Vision-Language Model

Model Description

Model Details

• Model Type: Vision-Language Model (CLIP-style)
• Total Parameters: 180,150,081 (180M)
• Base Architecture: Modified i3-Ethan (a SDPA version of Ethan) hybrid architecture
• Training Data: Midjourney Detailed Prompts dataset
• Model Dimensions: 768-dimensional embeddings
• Architecture Components:
  • Vision Encoder: ResNet-based CNN (16-layer deep)
  • Text Encoder: Hybrid RWKV + Transformer (12 RWKV layers + 4 Attention layers)
  • Contrastive Learning: Learnable temperature-scaled similarity

Architecture Overview

Vision Encoder (CNN-based):
├── Stem: Conv7x7 → BatchNorm → ReLU → MaxPool
├── Layer1: 3x ResBlocks (64 channels)
├── Layer2: 4x ResBlocks (128 channels)
├── Layer3: 6x ResBlocks (256 channels)
├── Layer4: 3x ResBlocks (512 channels)
└── AdaptiveAvgPool → Linear(512 → 768)

Text Encoder (Hybrid):
├── Token Embedding (vocab_size → 768)
├── Positional Embedding (77 positions)
├── 12x RWKV Blocks (parallel linear attention)
├── 4x Transformer Blocks (12 heads, multi-head attention)
└── Layer Norm → Take last token

Output:
└── Normalized embeddings → Contrastive loss

Key Features

1. Hybrid Text Processing

• RWKV Layers (12): Efficient linear-time attention for capturing sequential dependencies
• Transformer Layers (4): Full self-attention for complex reasoning
• Advantage: Combines O(n) efficiency of RWKV with the expressiveness of transformers

2. JIT-Optimized RWKV

• Custom torch.jit.script implementation of parallel RWKV attention
• Efficient time-mixing and channel-mixing mechanisms
• Learnable decay rates and position-dependent mixing

3. Contrastive Learning

• Bidirectional image-text matching
• Learnable temperature parameter (initialized to log(1/0.07))
• Symmetric cross-entropy loss

4. Efficient Vision Encoding

• ResNet-inspired architecture with 16 residual blocks
• Progressive channel expansion (64 → 128 → 256 → 512)
• Batch normalization and residual connections for stable training

Training Details

• Framework: PyTorch with mixed precision training
• Optimizer: AdamW (lr=5e-5)
• Batch Size: 32
• Sequence Length: 77 tokens (CLIP standard)
• Image Size: 224×224
• Normalization: ImageNet-style (mean=[0.48, 0.45, 0.40], std=[0.26, 0.26, 0.27])
• Training Steps: 2,000 iterations
• Validation: Every 100 steps

Training Dataset

Relationship to i3-Ethan-SDPA

Inherited from i3-Ethan-SDPA:

1. Hybrid RWKV-Attention Architecture: The core design philosophy of combining RWKV blocks (efficient sequential processing) with standard attention layers (complex reasoning)
2. JIT-Optimized RWKV Implementation: Same parallel linear attention kernel
3. Time-Mixing Mechanisms: Token-wise interpolation between current and previous states
4. Channel-Mixing FFN: Squared ReLU activation in feed-forward networks

Key Adaptations for Vision-Language Tasks:

1. Dual Encoder Design: Separate vision and text pathways (vs. single autoregressive decoder)
2. Contrastive Objective: Image-text matching (vs. next-token prediction)
3. Vision Processing: Added ResNet encoder for image understanding
4. Fixed-Length Embeddings: Pooled representations (vs. sequential generation)
5. Reduced Scale: 180M parameters focused on representation (vs. 200M for generation)

Use Cases

• Zero-shot image classification
• Image-text retrieval
• Visual search engines
• Multimodal embedding generation
• Cross-modal understanding tasks

Limitations

1. Training Scale: Trained on 2,000 iterations with a single dataset
2. Domain Specificity: Optimized for Midjourney-style synthetic images
3. Limited Context: 77-token text limit (CLIP standard)
4. No Fine-Grained Localization: Global image embeddings only
5. Checkpoint Availability: Best model saved based on validation loss

Ethical Considerations

• Synthetic Training Data: Model trained on AI-generated images may not generalize perfectly to real-world photos
• Bias Propagation: May inherit biases from the Midjourney dataset
• Content Safety: Should be evaluated for fairness across diverse demographics before deployment