Vfusion3d

--- license: cc-by-nc-2.0 pipeline_tag: image-to-3d library_name: transformers --- # [ECCV 2024] VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models [Porject page](https://junlinhan.github.io/projects/vfusion3d.html), [Paper link](https://arxiv.org/abs/2403.12034) VFusion3D is a large, feed-forward 3D generative model trained with a small amount of 3D data and a large volume of synthetic multi-view data. It is the first work exploring scalable 3D generative/reconstruction models as a step towards a 3D foundation. [VFusion3D: Learning Scalable 3D Generative Models from Video Diffusion Models](https://junlinhan.github.io/projects/vfusion3d.html)<br> [Junlin Han](https://junlinhan.github.io/), [Filippos Kokkinos](https://www.fkokkinos.com/), [Philip Torr](https://www.robots.ox.ac.uk/~phst/)<br> GenAI, Meta and TVG, University of Oxford<br> European Conference on Computer Vision (ECCV), 2024 ## News - [08.08.2024] [HF Demo](https://huggingface.co/spaces/facebook/VFusion3D) is available, big thanks to [Jade Choghari](https://github.com/jadechoghari)'s help for making it possible. - [25.07.2024] Release weights and inference code for VFusion3D. ## Quick Start Getting started with VFusion3D is super easy! 🤗 Here’s how you can use the model with Hugging Face: ### Install Dependencies (Optional) Depending on your needs, you may want to enable specific features like mesh generation or video rendering. We've got you covered with these additional packages: ```bash !pip --quiet install imageio[ffmpeg] PyMCubes trimesh rembg[gpu,cli] kiui ``` ### Load model directly ```python import torch from transformers import AutoModel, AutoProcessor # load the model and processor model = AutoModel.from_pretrained("jadechoghari/vfusion3d", trust_remote_code=True) processor = AutoProcessor.from_pretrained("jadechoghari/vfusion3d") # download and preprocess the image import requests from PIL import Image from io import BytesIO image_url = 'https://sm.ign.com/ign_nordic/cover/a/avatar-gen/avatar-generations_prsz.jpg' response = requests.get(image_url) image = Image.open(BytesIO(response.content)) # preprocess the image and get the source camera image, source_camera = processor(image) # generate planes (default output) output_planes = model(image, source_camera) print("Planes shape:", output_planes.shape) # generate a 3D mesh output_planes, m...

Source context: 42 downloads · 65 likes · Pipeline image-to-3d · Library transformers · Repo facebook/vfusion3d