ReArchitecture Multimodal RAG System Pipeline Journey

A multimodal AI system should be able to understand both text and video content.

Step 1 - Learn Gradio (UI) (30 mins)

Key Concepts:

• fn: The function wrapped by the UI.
• inputs: The Gradio components used for input (should match function arguments).
• outputs: The Gradio components used for output (should match return values).

• The component name as a string (e.g., "textbox")
• An instance of the component class (e.g., gr.Textbox())

Sharing Your Demo

Gradio Advanced Features

Gradio.Blocks

• Arrange components freely on the page.
• Handle multiple data flows.
• Use outputs as inputs for other components.
• Dynamically update components based on user interaction.

Gradio.ChatInterface

• Always set type="messages" in gr.ChatInterface.
• The default (type="tuples") is deprecated and will be removed in future versions.
• For more UI flexibility, use gr.ChatBot.
• gr.ChatInterface supports Markdown (not tested yet).

Step 2 - Learn Bridge Tower Embedding Model (Multimodal Learning) (15 mins)

Measuring Similarity

• Cosine Similarity → Measures how close images are in vector space (efficient & commonly used).
• Euclidean Distance → Uses cv2.NORM_L2 to compute similarity between two images.

Converting to 2D for Visualization

• UMAP reduces 512D embeddings to 2D for display purposes.

Preprocessing Videos for Multimodal RAG

Case 1: WEBVTT → Extracting Text Segments from Video

- Converts video + text into structured metadata.  
- Splits content into multiple segments.  

Case 2: Whisper (Small) → Video Only

- Extracts **audio** → `model.transcribe()`.  
- Applies `getSubs()` helper function to retrieve **WEBVTT** subtitles.  
- Uses **Case 1** processing.  

Case 3: LvLM → Video + Silent/Music Extraction

- Uses **Llava (LvLM model)** for **frame-based captioning**.  
- Encodes each frame as a **Base64 image**.  
- Extracts context and captions from video frames.  
- Uses **Case 1** processing.  

Step 4 - What is LLaVA?

Step 5 - what is a vector Store?

• Store and manage high-dimensional vector data efficiently
• Perform similarity-based searches where K=1 returns the most similar result
• In LanceDB specifically, store multiple data types: . Text content (captions) . Image file paths . Metadata . Vector embeddings

Gotchas and Solutions

Image Processing: When working with base64 encoded images, convert them to PIL.Image format before processing with BridgeTower
Model Selection: Using BridgeTowerForContrastiveLearning instead of PredictionGuard due to API access limitations
Model Size: BridgeTower model requires ~3.5GB download
Image Downloads: Some Flickr images may be unavailable; implement robust error handling
Token Decoding: BridgeTower contrastive learning model works with embeddings, not token predictions
Install from git+https://github.com/openai/whisper.git 

Install ffmepg using brew

```bash
    brew install ffmpeg
    brew link ffmpeg
```

    

Learning and Skills

Technical Skills:

Basic Machine learning and deep learning
Vector embeddings and similarity search
Multimodal data processing

Framework & Library Expertise:

Hugging Face Transformers
Gradio UI development
LangChain integration (Basic)
PyTorch basics
LanceDB vector storage

AI/ML Concepts:

Multimodal RAG system architecture
Vector embeddings and similarity search
Large Language Models (LLaVA)
Image-text pair processing
Dimensionality reduction techniques

Multimedia Processing:

Video frame extraction
Audio transcription (Whisper)
Image processing (PIL)
Base64 encoding/decoding
WebVTT handling

System Design:

Client-server architecture
API endpoint design
Data pipeline construction
Vector store implementation
Multimodal system integration

Hugging Face