You can deploy this model and start earning money today!

microsoft/
elem2design

Name: elem2design
Rating: 2.5 (86 reviews)

Views

No views yet

Model Details

Model Description

This model aims to compose user-provided graphic elements into a pleasing graphical design. It takes graphic elements (i.e., the images and texts) from users as input and generates the position, color and font information of each element as output.

Developed by: Jiawei Lin, Shizhao Sun, Danqing Huang, Ting Liu, Ji Li and Jiang Bian
Model type: Large Language Models
Language(s): Python
License: MIT
Finetuned from model: Llama-3.1-8B

Model Sources

Repository: https://github.com/microsoft/elem2design
Paper: https://arxiv.org/abs/2412.19712

Uses

Direct Use

Compose user-provided graphic elements (i.e., images and texts) into a pleasing graphic design.

Elem2Design is being shared with the research community to facilitate reproduction of our results and foster further research in this area.

Out-of-Scope Use

We do not recommend using Elem2Design in commercial or real-world applications without further testing and development. It is being released for research purposes.

Use in any manner that violates applicable laws or regulations.

Risks and Limitations

Elem2Design inherits any biases, errors, or omissions produced by its base model. Developers are advised to choose an appropriate base LLM/MLLM carefully, depending on the intended use case. 

Elem2Design uses the Llama model. See https://huggingface.co/meta-llama/Llama-3.1-8B to understand the capabilities and limitations of this model.  

As the model is fine-tuned on very specific data about design composition, it is unlikely to generate information other than position, color and font. However, this is possible. It is more likely to happen when instructions unrelated to graphic design composition, e.g., how has the social media influenced our daily life, are fed into the model.

Graphic designs generated by Elem2Design may not be technically accurate or meet user specifications in all cases. Users are responsible for assessing the acceptability of generated content for each intended use case.

Elem2Design was developed for research and experimental purposes. Further testing and validation are needed before considering its application in commercial or real-world scenarios.

Recommendations

Please only provide the images and texts that you want to show on the graphic design to the model.

Users are responsible for sourcing their content legally and ethically. This could include securing appropriate copy rights, ensuring consent for use of images of people, and/or the anonymization of data prior to use in research.  

How to Get Started with the Model

python llava/infer/infer.py \
      --model_name_or_path /path/to/model/checkpoint-xxxx \
      --data_path /path/to/data/test.json \
      --image_folder /path/to/crello_images \
      --output_dir /path/to/output_dir \
      --start_layer_index 0 \ 
      --end_layer_index 4

For more information, please visit our GitHub repo: https://github.com/microsoft/elem2design.

Training Details

Training Data

The training data is from an open-source dataset (https://huggingface.co/datasets/cyberagent/crello).

Training Procedure

Preprocessing

The training samples with more than 25 design elements are filtered out to maintain a limited sequence length and thereby improve training efficiency.

Training Hyperparameters

Learning rate: 2e-4
Global batch size: 128
Number of training steps: 7000
Rank and alpha of LoRA: 32 and 64

Speeds, Sizes, Times

Llama-3.1-8B: 8B parameters
CLIP ViT-Large-Patch14: 428M parameters

Evaluation

Testing Data

The testing data is from an open-source dataset (https://huggingface.co/datasets/cyberagent/crello).

Metrics

Overall metrics. We use a robust proxy model (https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-ov-hf) for comprehensive evaluation from five aspects: (i) design and layout, (ii) content relevance, (iii) typography and color, (iv) graphics and images, and (v) innovation and originality. We use the same prompts as presented in COLE [1].
Geometry-related metrics. These metrics focus purely on the geometric attributes of elements without considering their content, including element validity (Val), Overlap (Ove), Alignment (Ali) and underlay effectiveness (Undl, Unds) [2][3].

Results

We use prior work FlexDM [1] and prompting GPT-4o [2] as baselines. In comparison, Elem2Design demonstrates superior performance across nearly all metrics. For example, on overall metrics, Elem2Design achieves 8.08, 7.92, 8.00, 7.82 and 6.98 on the evaluated five aspects. For another example, regarding geometry-related metrics, Elem2Design and FlexDM achieve Ove score of 0.0865 and 0.3242 respectively, indicting that Elem2Design effectively addresses the overlap issue whereas FlexDM encounters difficulties in this area. See Table 1 for the complete evaluation in our paper (https://arxiv.org/pdf/2412.19712)

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Citation

@InProceedings{lin2024elements,
  title={From Elements to Design: A Layered Approach for Automatic Graphic Design Composition},
  author={Lin, Jiawei and Sun, Shizhao and Huang, Danqing and Liu, Ting and Li, Ji and Bian, Jiang},
  booktitle={CVPR},
  year={2025}
}

Model Card Contact

We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at Shizhao Sun, shizsu@microsoft.com.

If the team receives reports of undesired behavior or identifies issues independently, we will update this repository with appropriate mitigations. 