You can deploy this model and start earning money today!

wovenbytoyota-vai/
Woven-City-AI-Vision-Engine

Name: Woven-City-AI-Vision-Engine
Availability: InStock

Views

No views yet

Woven City AI Vision Engine

🧠 Model Summary

Woven City AI Vision Engine is a cutting-edge, 8B parameter long-context video modeling solution that enables efficient and high-quality video understanding. The model introduces a hierarchical compression method to process long/short videos without the typical quality degradation or high computational costs. This allows for a comprehensive analysis of videos, capturing both fine-grained details and overarching narratives.

✨ Key Features

Hierarchical Compression: Employs a sophisticated two-tier compression strategy. A fine-grained compressor that processes short-term temporal information, while a coarse-grained compressor handles long-term context.
Long-Context Understanding: Capable of processing and understanding videos of extended duration.
Efficient and High-Fidelity: Maintains high-fidelity video representation with a high compression ratio ensuring both efficiency and accuracy.
Cutting-edge Performance: Outperforms existing models in long-video question answering and excels in the video understanding benchmark MVBench with a score of 73.81%

🏆 MVBench Evaluation Leaderboard

Results confirmed as of 2025.09. Evaluated on the standard MVBench test set.

Rank	Model	Score
1	Woven City AI Vision Engine (ours, 8B)	73.81
2	Qwen3-VL-30B-A3B	71.87
3	Qwen3-Omni-30B-A3B	69.50
4	Video-CCAM-7B-v1.2	69.23
5	TimeMarker	67.425
6	InternVideo2-8B-HD-Chat-f16	67.25
7	Video-CCAM-9B-v1.1	64.6
8	Video-CCAM-4B-v1.1	62.8
9	VideoChat2_HD_mistral	62.3

⚙️ How to Use

This model is distributed as a Model Package on the Amazon Marketplace. To use it, you must first subscribe to the model and then deploy it to a SageMaker endpoint. The following steps guide you through deploying the model and performing inference for text, image, and video analysis. Please refer to the inference.ipynb notebook for using this model using Amazon Marketplace.

1. Deploy the Model to a SageMaker Endpoint

After subscribing to the model package on the AWS Marketplace, you can use the following Python code to create a SageMaker endpoint. You will need your AWS account's execution role and the Model Package ARN (Amazon Resource Name) from the subscription page.

Deployment Script

python

import sagemaker
from sagemaker import ModelPackage
from sagemaker.predictor import Predictor
from datetime import datetime

# --- Configuration ---
# Get your execution role
try:
    role = sagemaker.get_execution_role()
except ValueError:
    # If not in a SageMaker environment, specify the ARN directly
    role = "arn:aws:iam::YOUR_ACCOUNT_ID:role/YourSageMakerExecutionRole"

# Find your Model Package ARN on the AWS Marketplace subscription page
model_package_arn = "arn:aws:sagemaker:REGION:ACCOUNT_ID:model-package/YourModelPackageName" 

# Define deployment instance type and count
instance_type = "ml.g5.2xlarge" # Recommended instance type
instance_count = 1

# Create a unique endpoint name
endpoint_name = f"woven-city-ai-vision-engine-endpoint-{datetime.now().strftime('%H%M%S')}"

# --- Deployment ---
sagemaker_session = sagemaker.Session()

# Create a model package object
model_package = ModelPackage(
    role=role,
    model_package_arn=model_package_arn,
    sagemaker_session=sagemaker_session
)

# Deploy the model
print(f"Deploying endpoint '{endpoint_name}'... This may take 10-15 minutes.")
predictor = model_package.deploy(
    initial_instance_count=instance_count,
    instance_type=instance_type,
    endpoint_name=endpoint_name
)

print(f"Model deployed successfully to endpoint: {predictor.endpoint_name}")

2. Setup and Helper Functions

Once the endpoint is InService, set up your environment to connect to it. Define helper functions for encoding media and sending requests.

Connect to the Endpoint

python

import boto3
import json
import base64
import sagemaker
from sagemaker.predictor import Predictor

# Configure your AWS session if needed
# boto3.setup_default_session(profile_name = 'your-profile')
session = boto3.Session()
sagemaker_session = sagemaker.Session()
region = session.region_name

# Use the endpoint name from the deployment step
# endpoint_name = "your-sagemaker-endpoint-name" 

predictor = Predictor(
    endpoint_name=endpoint_name,
    sagemaker_session=sagemaker_session
)

print(f"Connected to endpoint: {predictor.endpoint_name}")

Helper Functions

python

def encode_image_to_base64(image_path):
    """Encode image file to base64 data URL"""
    with open(image_path, 'rb') as f:
        image_data = base64.b64encode(f.read()).decode('utf-8')
    
    if image_path.lower().endswith(('.jpg', '.jpeg')):
        mime_type = 'image/jpeg'
    elif image_path.lower().endswith('.png'):
        mime_type = 'image/png'
    else:
        mime_type = 'image/jpeg'
    
    return f"data:{mime_type};base64,{image_data}"

def encode_video_to_base64(video_path):
    """Encode video file to base64 data URL"""
    with open(video_path, 'rb') as f:
        video_data = base64.b64encode(f.read()).decode('utf-8')
    
    return f"data:video/mp4;base64,{video_data}"

def send_request(predictor, payload):
    """Send request to SageMaker endpoint with error handling"""
    try:
        payload_json = json.dumps(payload)
        
        response = predictor.predict(
            payload_json,
            initial_args={"ContentType": "application/json", "Accept": "application/json"}
        )
        
        if isinstance(response, bytes):
            response = response.decode('utf-8')
        
        if isinstance(response, str):
            response = json.loads(response)
            
        return response
        
    except Exception as e:
        print(f"An error occurred: {e}")
        return None

3. 📄 Text Analysis

You can use the model for standard text generation tasks by providing a text-only prompt.

Example Request

python

text_payload = {
    "instances": [{
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Explain the importance of workplace safety in manufacturing environments."
                }
            ]
        }],
        "parameters": {
            "max_new_tokens": 120,
            "temperature": 0.7,
            "do_sample": True
        }
    }]
}

print("=== Text Analysis ===")
response = send_request(predictor, text_payload)
if response:
    print("Generated Text:")
    print(response.get('generated_text', 'No response'))

Generated Text: Workplace safety in manufacturing environments is paramount for the well-being of employees and the smooth operation of the facility. It involves a range of measures, from the use of personal protective equipment to the implementation of safety protocols and training programs. Ensuring a safe workplace not only protects the physical health of the employees but also enhances productivity, reduces costs associated with accidents, and fosters a positive work environment.

4. 🖼️ Image Analysis

Provide an image along with a text prompt to perform visual question answering or description tasks.

Example Request

python

# Note: Replace 'sample1.jpg' with your image file path
image_path = "media/sample1.jpg" 

image_payload = {
    "instances": [{
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "What do you see in this picture."
                },
                {
                    "type": "image_url",
                    "image_url": {
                        "url": encode_image_to_base64(image_path),
                        "detail": "high"
                    }
                }
            ]
        }],
        "parameters": {
            "max_new_tokens": 150,
            "temperature": 0.6
        }
    }]
}

print("=== Image Analysis ===")
response = send_request(predictor, image_payload)
if response:
    print("Analysis:")
    print(response.get('generated_text', 'No response'))

Example Input (`sample1.jpg`)

The video showcases a breathtaking mountainous landscape. The scene is dominated by a majestic mountain in the background, its peak capped with snow, indicating a high altitude. The mountain's slopes are a mix of rocky terrain and patches of snow, suggesting a rugged and possibly challenging environment. In the foreground, there are several small lakes, their calm waters reflecting the surrounding scenery. The lakes are nestled among lush green forests, with trees of varying heights and densities. The sky above is clear and blue, adding to the serene and picturesque quality of the scene. The overall composition of the video highlights the natural beauty and tranquility of the landscape, with the mountain, lakes, and forest creating a harmonious and visually appealing scene.

5. 📹 Video Analysis

For video analysis, provide a video file and a prompt to ask questions about its content, summarize events, or identify actions.

Example Request

python

video_path = "media/sample2.mp4"

video_payload = {
    "instances": [{
        "messages": [{
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "Describe what happens in this video and identify any notable events or actions."
                },
                {
                    "type": "video_url",
                    "video_url": {
                        "url": encode_video_to_base64(video_path),
                        "fps": 2.0,
                        "max_frames": 16
                    }
                }
            ]
        }],
        "parameters": {
            "max_new_tokens": 180,
            "temperature": 0.6,
            "fps": 2.0,
            "max_num_frames": 16
        }
    }]
}

print("=== Video Analysis ===")
response = send_request(predictor, video_payload)
if response:
    print("Analysis:")
    print(response.get('generated_text', 'No response'))

Example Input (`sample.mp4`)

The video presents a sweeping aerial view of London's skyline, showcasing a variety of architectural styles and building heights. The sequence begins with a wide shot of the city, gradually panning in to reveal more details. The buildings are predominantly modern, with glass facades reflecting the sunlight, and some have distinctive shapes, such as the curved glass structure and the spire of the Shard. The color palette is dominated by the blue of the sky and the grey of the buildings, with occasional greenery on rooftops. There are no visible characters or movement, suggesting a focus on the city's static beauty. The light is bright and natural, indicating daytime with clear weather conditions. The video captures the essence of London's urban landscape, emphasizing its density and architectural diversity.

📝 Parameters Description

Parameter	Type	Default	Valid Range	Description & Notes
`max_new_tokens`	integer	50	1 – 1024	Max tokens generated beyond the prompt. Values above 1024 are silently clamped to 1024. The default of 50 is short — raise it for long outputs or responses will appear truncated. Higher values increase latency.
`do_sample`	boolean	true	—	Master switch for sampling. When `false`, decoding is deterministic (greedy / beam) and `temperature`, `top_p`, `top_k` are ignored.
`temperature`	float	0.6	0.0 – 2.0	Sampling randomness: lower = more focused, higher = more diverse. Only applies when `do_sample=true`. Out-of-range values are clamped.
`top_p`	float	0.95	0.1 – 1.0	Nucleus sampling threshold: sample from the smallest set with cumulative probability ≥ `top_p`. Only applies when `do_sample=true`. Clamped to range.
`top_k`	integer	null (disabled)	1 – 100	Restricts sampling to the top-k most likely tokens. Disabled by default; only applies when `do_sample=true` and explicitly set.
`num_beams`	integer	1	1 – 5	Beam search width for deterministic decoding. 1 = greedy. Larger = more thorough but slower. Values above 5 are clamped.
`fps`	float	1.0	0.1 – 10.0	Video frame sampling rate. Actual frames sampled = `fps × video_duration`, then capped by `max_num_frames`.
`max_num_frames`	integer	128	1 – 2048	Upper cap on sampled video frames. This is a ceiling, not a target — if `fps × duration` is smaller, fewer frames are used.

Reproducibility: for deterministic, repeatable output set do_sample=false. Sampling parameters are then ignored.

⚠️ Common Mistakes

Symptom	Cause	Fix
Output cut off mid-sentence	`max_new_tokens` left at the default of 50	Set it explicitly (e.g. 512)
Set `max_new_tokens` to 5000 but output is still short	Hard cap is 1024	1024 is the maximum
`temperature` / `top_p` / `top_k` have no effect	`do_sample=false`	Set `do_sample=true`
Video appears under-analyzed	Low `fps` produces few frames	Raise `fps`, not `max_num_frames`
`413` / payload too large (real-time)	Real-time platform limit is 6 MB	Keep raw media under ~4 MB, or use Batch Transform

Additional Resources

Amazon Marketplace — Amazon Marketplace
Visit our Amazon Bedrock Marketplace listing to deploy Woven City AI Vision Engine in your AWS account
InstVL — InstVL Dataset
InstVL is a large-scale dataset of images and videos designed to bridge the gap between holistic scene understanding and fine-grained, instance-level comprehension.
InstQA — InstQA Dataset
InstQA is a large-scale dataset of images and videos. The dataset contains dense instance-level captions and Visual Question Answers.
WTS Dataset — WTS: Woven Traffic Safety Dataset
A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understanding

🙏 Acknowledgements

This dataset is based on results obtained from a project, JPNP20017, subsidized by the New Energy and Industrial Technology Development Organization (NEDO).

wovenbytoyota-vai/Woven-City-AI-Vision-Engine

Woven City AI Vision Engine

🧠 Model Summary

✨ Key Features

🏆 MVBench Evaluation Leaderboard

⚙️ How to Use

1. Deploy the Model to a SageMaker Endpoint

Deployment Script

2. Setup and Helper Functions

Connect to the Endpoint

Helper Functions

3. 📄 Text Analysis

Example Request

4. 🖼️ Image Analysis

Example Request

Example Input (sample1.jpg)

5. 📹 Video Analysis

Example Request

Example Input (sample.mp4)

📝 Parameters Description

⚠️ Common Mistakes

Additional Resources

🙏 Acknowledgements

wovenbytoyota-vai/
Woven-City-AI-Vision-Engine

Example Input (`sample1.jpg`)

Example Input (`sample.mp4`)