Views
No views yet
npm i @huggingface/transformers1import {
2 AutoProcessor,
3 AutoModelForImageTextToText,
4 load_image,
5 TextStreamer,
6} from "@huggingface/transformers";
7
8// Load processor and model
9const model_id = "onnx-community/FastVLM-0.5B-ONNX";
10const processor = await AutoProcessor.from_pretrained(model_id);
11const model = await AutoModelForImageTextToText.from_pretrained(model_id, {
12 dtype: {
13 embed_tokens: "fp16",
14 vision_encoder: "q4",
15 decoder_model_merged: "q4",
16 },
17});
18
19// Prepare prompt
20const messages = [
21 {
22 role: "user",
23 content: "<image>Describe this image in detail.",
24 },
25];
26const prompt = processor.apply_chat_template(messages, {
27 add_generation_prompt: true,
28});
29
30// Prepare inputs
31const url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg";
32const image = await load_image(url);
33const inputs = await processor(image, prompt, {
34 add_special_tokens: false,
35});
36
37// Generate output
38const outputs = await model.generate({
39 ...inputs,
40 max_new_tokens: 512,
41 do_sample: false,
42 streamer: new TextStreamer(processor.tokenizer, {
43 skip_prompt: true,
44 skip_special_tokens: false,
45 // callback_function: (text) => { /* Do something with the streamed output */ },
46 }),
47});
48
49// Decode output
50const decoded = processor.batch_decode(
51 outputs.slice(null, [inputs.input_ids.dims.at(-1), null]),
52 { skip_special_tokens: true },
53);
54console.log(decoded[0]);The image depicts a vibrant and colorful scene featuring a variety of flowers and plants. The main focus is on a striking pink flower with a dark center, which appears to be a type of petunia. The petals are a rich, deep pink, and the flower has a classic, slightly ruffled appearance. The dark center of the flower is a contrasting color, likely a deep purple or black, which adds to the flower's visual appeal.
In the background, there are several other flowers and plants, each with their unique colors and shapes. To the left, there is a red flower with a bright, vivid hue, which stands out against the pink flower. The red flower has a more rounded shape and a lighter center, with petals that are a lighter shade of red compared to the pink flower.
To the right of the pink flower, there is a plant with red flowers, which are smaller and more densely packed. The red flowers are a deep, rich red color, and they have a more compact shape compared to the pink flower.
In the foreground, there is a green plant with a few leaves and a few small flowers. The leaves are a bright green color, and the flowers are a lighter shade of green, with a few petals that are slightly open.
Overall, the image is a beautiful representation of a garden or natural setting, with a variety of flowers and plants that are in full bloom. The colors are vibrant and the composition is well-balanced, with the pink flower in the center drawing the viewer's attention.