MeshGPT is a text-to-3D model based on an autoencoder (tokenizer) and a transformer to generate the tokens.
The autoencoder's purpose is to be able to translate 3D models into tokens which then the decoder part of it can convert back to 3D mesh.
For all purposes and definitions the autoencoder is the world first published 3D model tokenizer! (correct me if i'm wrong!)
Model Details
The autoencoder (tokenizer) is a relative small model using 50M parameters and the transformer model uses 184M parameters and the core is based on GPT2-small.
Due to hardware contraints it's trained using a codebook/vocabablity size of 2048.
Devoloped & trained by: Me with credits for MeshGPT codebase to Phil Wang
This model has been created without any sponsors or renting any GPU hardware, so it has a very limited capability in terms what it can generate.
It can handle fine single objects such as 'chair' or 'table' but more complex objects requires more training (see training dataset section).
The is also a problem with the face orientation since the triangles order was optimized for the model before training.
However this will be fixed in later versions.
I've only had access to the free tier GPU on kaggle so this model is only trained on 4k models with max 250 triangles.
The dataset contains total of 800 text labels so in terms what it can generate it's limited.
3D models was sourced from objaverse, shapenet and ModelNet40.
How it works:
MeshGPT uses an autoencoder which takes 3D mesh (has support for quads but not implemented in this model) then quantizes them into a codebook which can be used as tokens.
The second part of MeshGPT is the transformer that trains on the tokens generated by the autoencoder while cross-attending to a text embedding.
The final product is a tokenizer and a transformer that can input a text embedding and then autoregressive generate a 3D model based on the text input.
The tokens generated by the transformer can then be converted into 3D mesh using the autoencoder.
