GEN3C Setup Bundle (v2)
Updated based on the actual repository code. Key facts from the source:
- GEN3C runs on Linux only. Cosmos/Apex/transformer-engine require Linux.
On Windows you must use WSL2 (Ubuntu 22.04 or 24.04 recommended).
- Default output resolution is 704 x 1280. Pass
--height/--width
to change it; the model is most tested at 704x1280.
- Frame count must satisfy
(N - 1) % 120 == 0 (i.e. 121, 241, 361,
481, ...). The README's "121*N - 1" wording is wrong; the code is right.
- The conda env name is fixed at
cosmos-predict1 by the YAML file.
- Custom trajectory IS now supported via the included monkey-patch
module (gen3c_custom_trajectory.py). No GEN3C source modification needed.
Files
| File | Purpose |
|---|
setup_gen3c.sh | One-shot install (Linux/WSL2 only) following INSTALL.md exactly |
blender_export_camera.py | Run in Blender to export the camera trajectory as JSON, including OpenCV-convention w2c matrices |
gen3c_custom_trajectory.py | Monkey-patch module; loads Blender JSON and feeds it into GEN3C |
run_gen3c.py | CLI wrapper, supports both preset and custom-JSON trajectories |
Workflow
1. One-time install (1-3 hours, dominated by Apex compile and checkpoint download)
1export HF_TOKEN=hf_your_token_here
2chmod +x setup_gen3c.sh
3./setup_gen3c.sh
This follows the official INSTALL.md exactly:
- Clones the repo with submodules
- Creates
cosmos-predict1 conda env from the bundled YAML (Python 3.10, CUDA 12.4, gcc 12.4)
- Installs pip requirements
- Patches transformer-engine symlinks
- Installs transformer-engine 1.12.0
- Builds NVIDIA Apex from source (this is the slow step)
- Installs MoGe for depth estimation
- Authenticates Hugging Face
- Downloads the GEN3C-Cosmos-7B checkpoints
- Runs the bundled test_environment.py
2. Sanity check the install with the canned example
1conda activate cosmos-predict1
2cd ~/Projects/GEN3C
3export CUDA_HOME=$CONDA_PREFIX
4export PYTHONPATH=$(pwd)
5python cosmos_predict1/diffusion/inference/gen3c_single_image.py \
6 --checkpoint_dir checkpoints \
7 --input_image_path assets/diffusion/000000.png \
8 --video_save_name test_install \
9 --guidance 1 \
10 --foreground_masking
Verify the output video looks reasonable before doing anything custom.
3. Run a preset trajectory on your image
1python /path/to/run_gen3c.py \
2 --gen3c_dir $(pwd) \
3 --image /path/to/Surius.png \
4 --output_name surius_zoom_out \
5 --preset_trajectory zoom_out \
6 --movement_distance 0.5 \
7 --num_video_frames 121
This confirms GEN3C handles your specific image. If it works, move on.
4. Export your Blender camera
In Blender (Scripting workspace):
- Load the .blend with the 3DE-driven camera animation
- Set the camera as the active scene camera
- Open
blender_export_camera.py, run it
- A
camera_trajectory.json lands next to your .blend file
The JSON contains OpenCV-convention world-to-camera 4x4 matrices, which
match GEN3C's internal format.
5. Run with your custom trajectory
1python /path/to/run_gen3c.py \
2 --gen3c_dir $(pwd) \
3 --image /path/to/Surius.png \
4 --output_name surius_blender_cam \
5 --trajectory_json /path/to/camera_trajectory.json \
6 --num_video_frames 121
The wrapper:
- Reads the JSON for metadata
- Snaps the frame count to GEN3C's required
120*N + 1 pattern
- Sets up a monkey-patch via
gen3c_custom_trajectory.py
- Runs
gen3c_single_image.py with --trajectory none, which the patch
intercepts to inject the Blender-derived w2c matrices
How the custom trajectory injection works
GEN3C's pipeline:
gen3c_single_image.py calls MoGe to predict depth from the input image- Builds a 3D point cloud cache anchored at world-to-camera = identity
- Calls
generate_camera_trajectory() to get a sequence of w2c matrices
- Renders the cache from each new camera position
- Diffusion model fills in occluded regions and refines
gen3c_custom_trajectory.py replaces step 3 with:
- Load Blender w2c matrices from JSON
- Resample to match GEN3C's required frame count
- Re-anchor so frame 0 is identity (matching MoGe's coordinate system)
- Return as a (1, N, 4, 4) tensor in the format GEN3C expects
No GEN3C source files are modified, so upstream pulls still work.
Resolution and your hero shot
GEN3C generates at 704 x 1280 by default. For 1920 x 1080 delivery:
- Option A: Pass
--height 1080 --width 1920 and hope the model behaves
at non-tested resolutions. May produce artifacts.
- Option B (recommended): Generate at 704 x 1280, then upscale to
1920 x 1080 with FlashVSR, Topaz Video AI, RTX Video Super Resolution,
or tile diffusion at low denoise.
The upscaling pass is also where you'd add the "high-frequency detail"
that AI video models naturally lack.
VRAM expectations
GEN3C's README says max observed inference memory is ~43 GB with full
offloading. For a Blackwell RTX PRO 6000:
- Unsliced (~96 GB): runs comfortably, no
--offload_all needed
- 2x 48 GB vGPU slice: requires
--offload_all, will be slow
- Smaller slices: won't fit
If you hit OOM, add --offload_all to the run_gen3c.py command.
Caveats and known unknowns
- I have not run this end-to-end. The patch module is based on the
code in
cosmos_predict1/diffusion/inference/camera_utils.py and the
call site in gen3c_single_image.py. First run might surface issues
with coordinate conventions, intrinsics scale, or depth-to-trajectory
alignment that need tuning.
- If the Blender trajectory is re-anchored to identity at frame 0,
the first frame of your output should look almost identical to your
input image (because MoGe and the cache see the same view). If it
doesn't, the coordinate conversion in
blender_export_camera.py
may need adjustment.
- The intrinsics from your Blender JSON are currently not used; MoGe's
predicted intrinsics are used instead because GEN3C's pipeline tunes
depth and intrinsics together. If you need to override that, edit
the patch module to substitute the JSON intrinsics.
Troubleshooting
- Apex compilation fails: check your gcc version matches the env
YAML (gcc 12.4). On WSL2 with Ubuntu 22.04 you may need to install
build tools:
sudo apt install build-essential.
- Out of memory during Apex compile: Apex uses lots of RAM during
C++/CUDA build. Close other apps, or limit parallel jobs by setting
MAX_JOBS=4 before the pip install.
- HuggingFace 401 on checkpoint download: your token may need
access to gated NVIDIA models. Check the GEN3C-Cosmos-7B model card
on HuggingFace and accept any license agreements there.
- "no kernel image available" at runtime: Blackwell may need
PyTorch built with sm_90 or newer kernels. If the bundled PyTorch
doesn't support it, you may need a nightly PyTorch build.
- First frame looks wrong / scene flipped: the OpenCV conversion
in
blender_export_camera.py may need adjustment. Try removing the
_rebase_to_first_frame call in the patch module to see raw
trajectory behavior.
