Multimodal & Multilingual PDF Embedding Pipeline with Gemma and Vertex AI

• Multimodal Descriptions (via Gemma): Processes tables and images from PDFs, generating rich descriptive text in French using the open-source Gemma 3.4B-IT model, which runs locally on your machine/Colab GPU.
• Multilingual Text Embeddings (via Vertex AI): Leverages Google Cloud's text-multilingual-embedding-002 model for embeddings, supporting a wide range of languages.
• Structured Output: Stores embeddings and metadata (PDF source, page number, content type, links to extracted assets) in a comprehensive JSON format.

How it Works

1. PDF Parsing: Utilizes PyMuPDF to extract text blocks and images, and Camelot to accurately extract tabular data.
2. Content Separation: Distinguishes between plain text, tables, and non-table images.
3. Multimodal Description (for Tables & Images using Gemma):
   • For tables, the pipeline captures an image of the table and also uses its text representation.
   • For standalone images (e.g., graphs, charts), it captures the image.
   • These images (and optionally table text) are then passed to the Gemma 3.4B-IT model (via the gemma Python library) with specific prompts to generate rich, descriptive text in French. This step runs locally and does not incur direct API costs.
4. Multilingual Text Embedding (via Vertex AI):
   • The cleaned text content (original text chunks, or generated descriptions for tables/images) is then passed to the text-multilingual-embedding-002 model (via Vertex AI).
   • This model generates a high-dimensional embedding vector (768 dimensions) for each piece of content. This step connects to Google Cloud Vertex AI and will incur costs.
5. JSON Output: All generated embeddings, along with rich metadata (original PDF, page, content type, links to extracted assets), are compiled into a single JSON file.

Requirements & Setup

1. Google Cloud Project with Billing Enabled (for Text Embeddings):
   • CRITICAL: The text embedding generation step uses Google Cloud Vertex AI. This will incur costs on your Google Cloud Platform account. Ensure you have an active billing account linked to your project.
   • Enable the Vertex AI API.
2. Authentication for Google Cloud (for Text Embeddings):
   • The easiest way to run this in a Colab environment is using google.colab.auth.authenticate_user().
   • For local execution, ensure your Google Cloud SDK is configured and authenticated (gcloud auth application-default login).
3. Hardware Requirements (for Gemma):
   • Running the Gemma 3.4B-IT model requires a GPU with sufficient VRAM (e.g., a Colab T4 or V100 GPU, or a local GPU with at least ~8-10GB VRAM is recommended). If a GPU is not available, Gemma will likely run on CPU but will be significantly slower.

Local Setup

1. Clone the repository:
   bash

   1git clone https://huggingface.co/Anonymous1223334444/pdf-multimodal-multilingual-embedding-pipeline 2cd pdf-multimodal-multilingual-embedding-pipeline
2. Install Python dependencies:
   pip install -r requirements.txt
   System-level dependencies for Camelot/PyMuPDF (Linux/Colab): You might need to install these system packages for PyMuPDF and Camelot to function correctly.
   bash

   1# Update package list 2sudo apt-get update 3# Install Ghostscript (required by Camelot) 4sudo apt-get install -y ghostscript 5# Install python3-tk (required by some PyMuPDF functionalities) 6sudo apt-get install -y python3-tk 7# Install OpenCV (via apt, for camelot-py[cv]) 8sudo apt-get install -y libopencv-dev python3-opencv
   Note: If you are running on Windows or macOS, the installation steps for camelot-py might differ. Refer to the Camelot documentation for more details.
3. Set up Environment Variables (for Vertex AI Text Embeddings):
   bash

   1export GOOGLE_CLOUD_PROJECT="your-gcp-project-id" 2export VERTEX_AI_LOCATION="us-central1" # Or your preferred Vertex AI region (e.g., us-east4)
   Replace your-gcp-project-id and us-central1 with your actual Google Cloud Project ID and Vertex AI region.
4. Place your PDF files: Create a docs directory in the root of the repository and place your PDF documents inside it.

   pdf-multimodal-multilingual-embedding-pipeline/
   ├── docs/
   │   └── your_document.pdf
   └── another_document.pdf

5. Run the pipeline:
   python run_pipeline.py
   The generated embedding file (embeddings_statistiques_multimodal.json) and extracted assets will be saved in the output/ directory.

Google Colab Usage

1. Open a new Google Colab notebook.
2. Change runtime to GPU: Go to Runtime > Change runtime type and select T4 GPU or V100 GPU.
3. Install system and Python dependencies:
   python

   1!pip uninstall -y camelot camelot-py # Ensure clean install 2!pip install PyMuPDF 3!apt-get update 4!apt-get install -y ghostscript python3-tk libopencv-dev python3-opencv 5!pip install camelot-py[cv] google-cloud-aiplatform tiktoken pandas beautifulsoup4 Pillow gemma jax jaxlib numpy
4. Authenticate to Google Cloud (for Vertex AI):
   python

   1from google.colab import auth 2auth.authenticate_user()
5. Set your Google Cloud Project ID and Location:
   python

   1import os 2# Replace with your actual Google Cloud Project ID 3os.environ["GOOGLE_CLOUD_PROJECT"] = "YOUR_GCP_PROJECT_ID_HERE" 4# Set your preferred Vertex AI location (e.g., "us-central1", "us-east4") 5os.environ["VERTEX_AI_LOCATION"] = "us-central1" 6 7# Critical: Adjust JAX memory allocation for Gemma 8os.environ["XLA_PYTHON_CLIENT_MEM_FRACTION"]="1.00"
6. Upload your PDF files: You can use the Colab file upload feature or mount Google Drive. Ensure your PDFs are in a directory named docs within /content/.
   python

   1# Example for uploading 2from google.colab import files 3import os 4from pathlib import Path 5 6PDF_DIRECTORY = Path("/content/docs") 7PDF_DIRECTORY.mkdir(parents=True, exist_ok=True) 8uploaded = files.upload() 9for filename in uploaded.keys(): 10 os.rename(filename, PDF_DIRECTORY / filename)
7. Copy and paste the code from src/pdf_processor.py, src/embedding_utils.py and run_pipeline.py into Colab cells and execute. Make sure to execute embedding_utils.py content first, then pdf_processor.py content, then run_pipeline.py content, or combine them logically into your notebook.

Output

• embeddings_statistiques_multimodal.json: A JSON file containing all generated embeddings and their metadata.
• output/extracted_graphs/: Directory containing extracted images (PNG format).
• output/extracted_tables/: Directory containing HTML representations of extracted tables.

Example embeddings_statistiques_multimodal.json Entry

Acknowledgments

• Gemma AI
• Google AI Gemini Models
• PyMuPDF
• Camelot
• Tiktoken
• Pandas
• BeautifulSoup