You can deploy this model and start earning money today!

internlm/
internlm-xcomposer2d5-7b-chat

Name: internlm-xcomposer2d5-7b-chat
Availability: InStock

pytorch

internlm2

visual-question-answering

custom_code

2501.12368

other

Views

No views yet

InternLM-XComposer-2.5-Chat

💻Github Repo

Paper

InternLM-XComposer2.5-Chat is a chat model trained on internlm/internlm-xcomposer2d5-7b, offers improved multi-modal instruction following and open-ended dialogue capabilities.

Import from Transformers

To load the InternLM-XComposer2-2d5-Chat model using Transformers, use the following code:

python

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
ckpt_path = "internlm/internlm-xcomposer2d5-7b-chat"
tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True).cuda()
# Set `torch_dtype=torch.floatb16` to load model in bfloat16, otherwise it will be loaded as float32 and might cause OOM Error.
model = AutoModelForCausalLM.from_pretrained(ckpt_path, torch_dtype=torch.bfloat16, trust_remote_code=True).cuda()
model = model.eval()

Quickstart

We provide a simple example to show how to use InternLM-XComposer2.5 with 🤗 Transformers.

Video Understanding

python

import torch
from transformers import AutoModel, AutoTokenizer

torch.set_grad_enabled(False)

# init model and tokenizer
model = AutoModel.from_pretrained('internlm/internlm-xcomposer2d5-7b-chat', torch_dtype=torch.bfloat16, trust_remote_code=True).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained('internlm/internlm-xcomposer2d5-7b-chat', trust_remote_code=True)
model.tokenizer = tokenizer

query = 'Here are some frames of a video. Describe this video in detail'
image = ['./examples/liuxiang.mp4',]
with torch.autocast(device_type='cuda', dtype=torch.float16):
    response, his = model.chat(tokenizer, query, image, do_sample=False, num_beams=3, use_meta=True)
print(response)
# The video begins with a man in a red and yellow uniform standing on the starting line of a track, preparing to compete in the 110-meter hurdles at the Athens 2004 Olympic Games. He is identified as Liu Xiang, a Chinese athlete, and his bib number is 1363. The scene is set in a stadium filled with spectators, indicating the significance of the event.
# As the race begins, all the athletes start running, but Liu Xiang quickly takes the lead. However, he encounters a hurdle and knocks it over. Despite this setback, he quickly recovers and continues to run. The race is intense, with athletes from various countries competing fiercely. In the end, Liu Xiang emerges as the winner with a time of 12.91 seconds, securing the gold medal for China.
# The video then transitions to a slow-motion replay of the race, focusing on Liu Xiang's performance and the knockdown of the hurdle. This allows viewers to appreciate the skill and determination of the athlete.
# Following the race, Liu Xiang is seen lying on the track, possibly exhausted from the intense competition. He then stands up and begins to celebrate his victory, waving his arms in the air and running around the track. The crowd cheers and celebrates with him, creating a joyful atmosphere.
# The video concludes with a replay of Liu Xiang's gold medal-winning moment, emphasizing the significance of his achievement at the Athens 2004 Olympic Games.
# Throughout the video, the Olympic logo is prominently displayed, reminding viewers of the global significance of the event and the athletes' dedication and perseverance in their pursuit of victory.

query = 'tell me the athlete code of Liu Xiang'
image = ['./examples/liuxiang.mp4',]
with torch.autocast(device_type='cuda', dtype=torch.float16):
    response, _ = model.chat(tokenizer, query, image, history=his, do_sample=False, num_beams=3, use_meta=True)
print(response)
# The athlete code of Liu Xiang is 1363.

Multi-Image Mutli-Tune Dialog

python

import torch
from transformers import AutoModel, AutoTokenizer

torch.set_grad_enabled(False)

# init model and tokenizer
model = AutoModel.from_pretrained('internlm/internlm-xcomposer2d5-7b-chat', torch_dtype=torch.bfloat16, trust_remote_code=True).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained('internlm/internlm-xcomposer2d5-7b-chat', trust_remote_code=True)
model.tokenizer = tokenizer

query = 'Image1 <ImageHere>; Image2 <ImageHere>; Image3 <ImageHere>; I want to buy a car from the three given cars, analyze their advantages and weaknesses one by one'
image = ['./examples/cars1.jpg',
        './examples/cars2.jpg',
        './examples/cars3.jpg',]
with torch.autocast(device_type='cuda', dtype=torch.float16):
    response, his = model.chat(tokenizer, query, image, do_sample=False, num_beams=3, use_meta=True)
print(response)
# Certainly! Let's analyze the advantages and disadvantages of each car:
# 
# 1. **Mercedes-Benz G-Class (SUV)**:
#    - **Advantages**:
#      - **Off-Road Capability**: The G-Class is renowned for its off-road prowess. It can handle a variety of terrains, making it ideal for adventurous driving.
#      - **Reliability**: Mercedes-Benz is known for producing reliable vehicles, and the G-Class is no exception. It has a reputation for durability and longevity.
#      - **Luxury Features**: As a Mercedes-Benz, the G-Class comes with a host of luxury features, including high-quality materials and advanced technology.
#    - **Disadvantages**:
#      - **Fuel Efficiency**: The G-Class is not known for its fuel efficiency. It consumes a significant amount of gasoline, which can be a disadvantage for those concerned with fuel economy.
#      - **Size and Weight**: The G-Class is large and heavy, which can affect its handling and maneuverability, especially in urban environments.
#      - **Cost**: The G-Class is generally more expensive compared to other SUVs, which can be a deterrent for some buyers.
# 
# 2. **Bugatti Chiron (Sports Car)**:
#    - **Advantages**:
#      - **Performance**: The Bugatti Chiron is one of the fastest production cars available. It boasts impressive acceleration and top speed, making it a thrilling driving experience.
#      - **Design**: The Chiron has a sleek and futuristic design that is both aesthetically pleasing and aerodynamically efficient.
#      - **Status Symbol**: Owning a Bugatti is often seen as a status symbol, attracting attention and admiration.
#    - **Disadvantages**:
#      - **Cost**: The Bugatti Chiron is one of the most expensive cars in the world, making it out of reach for many potential buyers.
#      - **Maintenance**: Bugatti vehicles require specialized maintenance, which can be costly and difficult to find.
#     - **Road Legality**: In some regions, the Chiron may not be road-legal due to its high top speed and powerful engine.
# 
# 3. **BMW i Vision Dynamics (Electric Concept Car)**:
#    - **Advantages**:
#      - **Environmental Impact**: As an electric vehicle, the i Vision Dynamics has a lower environmental impact compared to traditional gasoline-powered cars.
#      - **Design**: The i Vision Dynamics has a unique and futuristic design that sets it apart from conventional cars.
#      - **Performance**: Despite being an electric vehicle, the i Vision Dynamics boasts impressive performance, including a top speed of over 300 km/h.
#    - **Disadvantages**:
#      - **Cost**: Like the Bugatti Chiron, the i Vision Dynamics is an expensive concept car, making it inaccessible to most buyers.
#      - **Range**: Electric vehicles typically have a limited range compared to gasoline-powered cars, which can be a concern for long-distance travel.
#      - **Infrastructure**: The availability of charging stations can vary widely by region, which can limit the practicality of owning an electric car.

# In summary, the Mercedes-Benz G-Class is a reliable and capable off-road vehicle with luxury features, but it may not be the most fuel-efficient or affordable option. The Bugatti Chiron offers unparalleled performance and status, but it is extremely expensive and requires specialized maintenance. The BMW i Vision Dynamics is an innovative and environmentally friendly concept car with impressive performance, but it is also costly and limited by range and charging infrastructure. Each car has its own unique set of advantages and disadvantages, and the choice ultimately depends on individual preferences and priorities.

query = 'Image4 <ImageHere>; How about the car in Image4'
image.append('./examples/cars4.jpg')        
with torch.autocast(device_type='cuda', dtype=torch.float16):
    response, _ = model.chat(tokenizer, query, image, do_sample=False, num_beams=3, history= his, use_meta=True)
print(response)
# The car in Image4 is a **Ferrari LaFerrari**. The LaFerrari is a hybrid supercar produced by Ferrari, combining a V12 engine with an electric motor. Here's a detailed analysis of its advantages and disadvantages:
# 
# ### **Advantages:**
# 1. **Performance**: The LaFerrari is one of the fastest production cars available, with a top speed of over 217 mph (349 km/h). It boasts impressive acceleration and handling, making it a thrilling driving experience.
# 2. **Design**: The LaFerrari has a distinctive and aggressive design that sets it apart from other supercars. Its aerodynamic features and sleek lines contribute to its performance and visual appeal.
# 3. **Hybrid Technology**: The LaFerrari uses a hybrid powertrain, combining a 6.3-liter V12 engine with an electric motor. This hybrid system provides a balance of power and efficiency, reducing emissions compared to traditional gasoline engines.
# 4. **Status Symbol**: Owning a LaFerrari is often seen as a status symbol, attracting attention and admiration. It represents a pinnacle of automotive engineering and luxury.
# 5. **Reliability**: Ferrari is known for producing high-quality, reliable vehicles. The LaFerrari benefits from the brand's reputation for excellence in engineering and craftsmanship.

### **Disadvantages:**
# 1. **Cost**: The LaFerrari is one of the most expensive cars in the world, making it inaccessible to most potential buyers. Its high price can be a significant deterrent.
# 2. **Maintenance**: Ferrari vehicles require specialized maintenance, which can be costly and difficult to find. The hybrid system may also add to the complexity and expense of servicing the car.
# 3. **Road Legality**: In some regions, the LaFerrari may not be road-legal due to its high top speed and powerful engine. This can limit its usability and appeal.
# 4. **Fuel Efficiency**: Despite the hybrid system, the LaFerrari consumes a significant amount of fuel, which can be a disadvantage for those concerned with fuel economy.
# 5. **Size and Weight**: The LaFerrari is a large and heavy vehicle, which can affect its handling and maneuverability, especially in urban environments.

# In summary, the Ferrari LaFerrari is a high-performance hybrid supercar with a distinctive design and impressive capabilities. However, its high cost, specialized maintenance requirements, and limited road legality can be significant disadvantages for some buyers. The LaFerrari is best suited for those who prioritize performance, luxury, and status over practicality and affordability.

High Resolution Image Understanding

python

import torch
from transformers import AutoModel, AutoTokenizer

torch.set_grad_enabled(False)

# init model and tokenizer
model = AutoModel.from_pretrained('internlm/internlm-xcomposer2d5-7b-chat', torch_dtype=torch.bfloat16, trust_remote_code=True).cuda().eval()
tokenizer = AutoTokenizer.from_pretrained('internlm/internlm-xcomposer2d5-7b-chat', trust_remote_code=True)
model.tokenizer = tokenizer

query = 'Analyze the given image in a detail manner'
image = ['./examples/dubai.png']
with torch.autocast(device_type='cuda', dtype=torch.float16):
    response, _ = model.chat(tokenizer, query, image, do_sample=False, num_beams=3, use_meta=True)
print(response)
# The image is an infographic titled "Amazing Facts About Dubai." Here's a detailed analysis of its content:
# 
# 1. **Title and Introduction:**
#    - The title is prominently displayed at the top of the image in bold, blue letters.
#    - The image features a colorful skyline of Dubai, highlighting the city's modern architecture.
# 
# 2. **Facts About Palm Jumeirah:**
#    - Palm Jumeirah is the largest artificial island and is visible from space.
#    - In 1968, there were only 1.5 million cars in Dubai.
#
# 3. **Dubai's Gold Chain:**
#    - Dubai has the world's largest Gold Chain, which is 4.2 km long.
#    - 7 out of the 10 tallest hotels in the world are located in Dubai.
#
# 4. **Crime Rate and Income Tax:**
#    - The crime rate is near 0%.
#    - The income tax rate is 0%.
#
# 5. **Dubai Mall:**
#    - Dubai Mall is the largest shopping mall in the world with 1200 stores.
#    - 17% of the population is Emirati, and 83% are immigrants.
#
# 6. **Dubai's Address System:**
#    - Dubai has no standard address system, with no zip codes, area codes, or postal services.
#
# 7. **Dispense Gold:**
#    - Dubai is building a climate-controlled City, 2.25 times as big as Monaco.
#    - The Royal Suite at Burj Al Arab is $24,000 per night.
#
# 8. **License and Billionaires:**
#    - You need a license to drink alcohol even at home.
#    - The net worth of the four listed billionaires is roughly equal to the GDP of Honduras.
#
# 9. **Sources:**
#    - The infographic cites sources from Wikipedia, Forbes, Gulf News, and The Guardian.
#
# 10. **Design and Compilation:**
#     - The image is designed and compiled by FMEXtensions, a company based in the United Arab Emirates.
#
# The infographic uses a combination of text, icons, and images to convey interesting facts about Dubai, emphasizing its modernity, wealth, and unique features.

Open Source License

The code is licensed under Apache-2.0, while model weights are fully open for academic research and also allow free commercial usage. To apply for a commercial license, please fill in the application form (English)/申请表（中文）. For other questions or collaborations, please contact internlm@pjlab.org.cn.