Spaces:
Running
on
Zero
Running
on
Zero
root
commited on
Commit
·
6c84977
1
Parent(s):
9aad643
first commit
Browse filesThis view is limited to 50 files because it contains too many changes.
See raw diff
- .gitattributes +1 -0
- README.md +8 -6
- app.py +408 -0
- assets/example_image/T.png +3 -0
- assets/example_image/typical_building_building.png +3 -0
- assets/example_image/typical_building_castle.png +3 -0
- assets/example_image/typical_building_colorful_cottage.png +3 -0
- assets/example_image/typical_building_maya_pyramid.png +3 -0
- assets/example_image/typical_building_mushroom.png +3 -0
- assets/example_image/typical_building_space_station.png +3 -0
- assets/example_image/typical_creature_dragon.png +3 -0
- assets/example_image/typical_creature_elephant.png +3 -0
- assets/example_image/typical_creature_furry.png +3 -0
- assets/example_image/typical_creature_quadruped.png +3 -0
- assets/example_image/typical_creature_robot_crab.png +3 -0
- assets/example_image/typical_creature_robot_dinosour.png +3 -0
- assets/example_image/typical_creature_rock_monster.png +3 -0
- assets/example_image/typical_humanoid_block_robot.png +3 -0
- assets/example_image/typical_humanoid_dragonborn.png +3 -0
- assets/example_image/typical_humanoid_dwarf.png +3 -0
- assets/example_image/typical_humanoid_goblin.png +3 -0
- assets/example_image/typical_humanoid_mech.png +3 -0
- assets/example_image/typical_misc_crate.png +3 -0
- assets/example_image/typical_misc_fireplace.png +3 -0
- assets/example_image/typical_misc_gate.png +3 -0
- assets/example_image/typical_misc_lantern.png +3 -0
- assets/example_image/typical_misc_magicbook.png +3 -0
- assets/example_image/typical_misc_mailbox.png +3 -0
- assets/example_image/typical_misc_monster_chest.png +3 -0
- assets/example_image/typical_misc_paper_machine.png +3 -0
- assets/example_image/typical_misc_phonograph.png +3 -0
- assets/example_image/typical_misc_portal2.png +3 -0
- assets/example_image/typical_misc_storage_chest.png +3 -0
- assets/example_image/typical_misc_telephone.png +3 -0
- assets/example_image/typical_misc_television.png +3 -0
- assets/example_image/typical_misc_workbench.png +3 -0
- assets/example_image/typical_vehicle_biplane.png +3 -0
- assets/example_image/typical_vehicle_bulldozer.png +3 -0
- assets/example_image/typical_vehicle_cart.png +3 -0
- assets/example_image/typical_vehicle_excavator.png +3 -0
- assets/example_image/typical_vehicle_helicopter.png +3 -0
- assets/example_image/typical_vehicle_locomotive.png +3 -0
- assets/example_image/typical_vehicle_pirate_ship.png +3 -0
- assets/example_image/weatherworn_misc_paper_machine3.png +3 -0
- assets/example_multi_image/character_1.png +3 -0
- assets/example_multi_image/character_2.png +3 -0
- assets/example_multi_image/character_3.png +3 -0
- assets/example_multi_image/mushroom_1.png +3 -0
- assets/example_multi_image/mushroom_2.png +3 -0
- assets/example_multi_image/mushroom_3.png +3 -0
.gitattributes
CHANGED
@@ -33,6 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
|
|
36 |
*.png filter=lfs diff=lfs merge=lfs -text
|
37 |
*.whl filter=lfs diff=lfs merge=lfs -text
|
38 |
*.jpg filter=lfs diff=lfs merge=lfs -text
|
|
|
33 |
*.zip filter=lfs diff=lfs merge=lfs -text
|
34 |
*.zst filter=lfs diff=lfs merge=lfs -text
|
35 |
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
36 |
+
wheels/nvdiffrast-0.3.3-cp310-cp310-linux_x86_64.whl filter=lfs diff=lfs merge=lfs -text
|
37 |
*.png filter=lfs diff=lfs merge=lfs -text
|
38 |
*.whl filter=lfs diff=lfs merge=lfs -text
|
39 |
*.jpg filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
@@ -1,14 +1,16 @@
|
|
1 |
---
|
2 |
-
title:
|
3 |
-
emoji:
|
4 |
-
colorFrom:
|
5 |
-
colorTo:
|
6 |
sdk: gradio
|
7 |
-
sdk_version: 5.
|
8 |
app_file: app.py
|
9 |
pinned: false
|
10 |
license: mit
|
11 |
-
short_description:
|
12 |
---
|
13 |
|
14 |
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
|
|
|
|
|
|
1 |
---
|
2 |
+
title: TRELLIS - Multiple Imagen a 3D
|
3 |
+
emoji: 🚀
|
4 |
+
colorFrom: indigo
|
5 |
+
colorTo: yellow
|
6 |
sdk: gradio
|
7 |
+
sdk_version: 5.29.0
|
8 |
app_file: app.py
|
9 |
pinned: false
|
10 |
license: mit
|
11 |
+
short_description: Scalable and Versatile 3D Generation from images
|
12 |
---
|
13 |
|
14 |
Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
|
15 |
+
|
16 |
+
Paper: https://huggingface.co/papers/2412.01506
|
app.py
ADDED
@@ -0,0 +1,408 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
import gradio as gr
|
2 |
+
import spaces
|
3 |
+
import os
|
4 |
+
import shutil
|
5 |
+
os.environ['SPCONV_ALGO'] = 'native'
|
6 |
+
from typing import *
|
7 |
+
import torch
|
8 |
+
import numpy as np
|
9 |
+
import imageio
|
10 |
+
from easydict import EasyDict as edict
|
11 |
+
from PIL import Image
|
12 |
+
from trellis.pipelines import TrellisImageTo3DPipeline
|
13 |
+
from trellis.representations import Gaussian, MeshExtractResult
|
14 |
+
from trellis.utils import render_utils, postprocessing_utils
|
15 |
+
import requests
|
16 |
+
import base64
|
17 |
+
import io
|
18 |
+
import tempfile
|
19 |
+
MAX_SEED = np.iinfo(np.int32).max
|
20 |
+
TMP_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'tmp')
|
21 |
+
os.makedirs(TMP_DIR, exist_ok=True)
|
22 |
+
|
23 |
+
NODE_SERVER_UPLOAD_URL = "https://viverse-backend.onrender.com/api/upload-rigged-model"
|
24 |
+
|
25 |
+
# Funciones auxiliares
|
26 |
+
def start_session(req: gr.Request):
|
27 |
+
user_dir = os.path.join(TMP_DIR, str(req.session_hash))
|
28 |
+
os.makedirs(user_dir, exist_ok=True)
|
29 |
+
|
30 |
+
def end_session(req: gr.Request):
|
31 |
+
user_dir = os.path.join(TMP_DIR, str(req.session_hash))
|
32 |
+
shutil.rmtree(user_dir)
|
33 |
+
|
34 |
+
def preprocess_images(images: List[Tuple[Image.Image, str]]) -> List[Image.Image]:
|
35 |
+
images = [image[0] for image in images]
|
36 |
+
processed_images = [pipeline.preprocess_image(image) for image in images]
|
37 |
+
return processed_images
|
38 |
+
|
39 |
+
def pack_state(gs: Gaussian, mesh: MeshExtractResult) -> dict:
|
40 |
+
return {
|
41 |
+
'gaussian': {
|
42 |
+
**gs.init_params,
|
43 |
+
'_xyz': gs._xyz.cpu().numpy(),
|
44 |
+
'_features_dc': gs._features_dc.cpu().numpy(),
|
45 |
+
'_scaling': gs._scaling.cpu().numpy(),
|
46 |
+
'_rotation': gs._rotation.cpu().numpy(),
|
47 |
+
'_opacity': gs._opacity.cpu().numpy(),
|
48 |
+
},
|
49 |
+
'mesh': {
|
50 |
+
'vertices': mesh.vertices.cpu().numpy(),
|
51 |
+
'faces': mesh.faces.cpu().numpy(),
|
52 |
+
},
|
53 |
+
}
|
54 |
+
|
55 |
+
def unpack_state(state: dict) -> Tuple[Gaussian, edict]:
|
56 |
+
gs = Gaussian(
|
57 |
+
aabb=state['gaussian']['aabb'],
|
58 |
+
sh_degree=state['gaussian']['sh_degree'],
|
59 |
+
mininum_kernel_size=state['gaussian']['mininum_kernel_size'],
|
60 |
+
scaling_bias=state['gaussian']['scaling_bias'],
|
61 |
+
opacity_bias=state['gaussian']['opacity_bias'],
|
62 |
+
scaling_activation=state['gaussian']['scaling_activation'],
|
63 |
+
)
|
64 |
+
gs._xyz = torch.tensor(state['gaussian']['_xyz'], device='cuda')
|
65 |
+
gs._features_dc = torch.tensor(state['gaussian']['_features_dc'], device='cuda')
|
66 |
+
gs._scaling = torch.tensor(state['gaussian']['_scaling'], device='cuda')
|
67 |
+
gs._rotation = torch.tensor(state['gaussian']['_rotation'], device='cuda')
|
68 |
+
gs._opacity = torch.tensor(state['gaussian']['_opacity'], device='cuda')
|
69 |
+
mesh = edict(
|
70 |
+
vertices=torch.tensor(state['mesh']['vertices'], device='cuda'),
|
71 |
+
faces=torch.tensor(state['mesh']['faces'], device='cuda'),
|
72 |
+
)
|
73 |
+
return gs, mesh
|
74 |
+
|
75 |
+
def get_seed(randomize_seed: bool, seed: int) -> int:
|
76 |
+
return np.random.randint(0, MAX_SEED) if randomize_seed else seed
|
77 |
+
|
78 |
+
@spaces.GPU
|
79 |
+
def image_to_3d(
|
80 |
+
multiimages: List[Tuple[Image.Image, str]],
|
81 |
+
seed: int,
|
82 |
+
ss_guidance_strength: float,
|
83 |
+
ss_sampling_steps: int,
|
84 |
+
slat_guidance_strength: float,
|
85 |
+
slat_sampling_steps: int,
|
86 |
+
multiimage_algo: Literal["multidiffusion", "stochastic"],
|
87 |
+
req: gr.Request,
|
88 |
+
) -> Tuple[dict, str]:
|
89 |
+
user_dir = os.path.join(TMP_DIR, str(req.session_hash))
|
90 |
+
outputs = pipeline.run_multi_image(
|
91 |
+
[image[0] for image in multiimages],
|
92 |
+
seed=seed,
|
93 |
+
formats=["gaussian", "mesh"],
|
94 |
+
preprocess_image=False,
|
95 |
+
sparse_structure_sampler_params={
|
96 |
+
"steps": ss_sampling_steps,
|
97 |
+
"cfg_strength": ss_guidance_strength,
|
98 |
+
},
|
99 |
+
slat_sampler_params={
|
100 |
+
"steps": slat_sampling_steps,
|
101 |
+
"cfg_strength": slat_guidance_strength,
|
102 |
+
},
|
103 |
+
mode=multiimage_algo,
|
104 |
+
)
|
105 |
+
video = render_utils.render_video(outputs['gaussian'][0], num_frames=120)['color']
|
106 |
+
video_geo = render_utils.render_video(outputs['mesh'][0], num_frames=120)['normal']
|
107 |
+
video = [np.concatenate([video[i], video_geo[i]], axis=1) for i in range(len(video))]
|
108 |
+
video_path = os.path.join(user_dir, 'sample.mp4')
|
109 |
+
imageio.mimsave(video_path, video, fps=15)
|
110 |
+
state = pack_state(outputs['gaussian'][0], outputs['mesh'][0])
|
111 |
+
torch.cuda.empty_cache()
|
112 |
+
return state, video_path
|
113 |
+
|
114 |
+
@spaces.GPU(duration=90)
|
115 |
+
def extract_glb(
|
116 |
+
state: dict,
|
117 |
+
mesh_simplify: float,
|
118 |
+
texture_size: int,
|
119 |
+
req: gr.Request,
|
120 |
+
) -> Tuple[str, str]:
|
121 |
+
user_dir = os.path.join(TMP_DIR, str(req.session_hash))
|
122 |
+
gs, mesh = unpack_state(state)
|
123 |
+
glb = postprocessing_utils.to_glb(gs, mesh, simplify=mesh_simplify, texture_size=texture_size, verbose=False)
|
124 |
+
glb_path = os.path.join(user_dir, 'sample.glb')
|
125 |
+
glb.export(glb_path)
|
126 |
+
torch.cuda.empty_cache()
|
127 |
+
return glb_path, glb_path
|
128 |
+
|
129 |
+
@spaces.GPU(duration=180)
|
130 |
+
def generate_model_from_images_and_upload(
|
131 |
+
image_inputs: List[Dict[str, Any]],
|
132 |
+
input_type: str,
|
133 |
+
seed_val: int,
|
134 |
+
ss_guidance_strength_val: float,
|
135 |
+
ss_sampling_steps_val: int,
|
136 |
+
slat_guidance_strength_val: float,
|
137 |
+
slat_sampling_steps_val: int,
|
138 |
+
multiimage_algo_val: str,
|
139 |
+
mesh_simplify_val: float,
|
140 |
+
texture_size_val: int,
|
141 |
+
model_description: str,
|
142 |
+
req: gr.Request
|
143 |
+
) -> str:
|
144 |
+
user_dir = os.path.join(TMP_DIR, str(req.session_hash))
|
145 |
+
os.makedirs(user_dir, exist_ok=True)
|
146 |
+
|
147 |
+
# --- DEBUG LOGS ---
|
148 |
+
print(f"Python DEBUG: Raw image_inputs (as received by function): {image_inputs}")
|
149 |
+
print(f"Python DEBUG: Type of image_inputs: {type(image_inputs)}")
|
150 |
+
if isinstance(image_inputs, list):
|
151 |
+
print(f"Python DEBUG: Length of image_inputs list: {len(image_inputs)}")
|
152 |
+
if len(image_inputs) > 0 and isinstance(image_inputs[0], dict):
|
153 |
+
print(f"Python DEBUG: First element of image_inputs (should be a dict): {image_inputs[0]}")
|
154 |
+
print(f"Python DEBUG: Type of first element: {type(image_inputs[0])}")
|
155 |
+
print(f"Python DEBUG: Received input_type from Node.js: '{input_type}'") # Should always be 'url' now
|
156 |
+
# --- END DEBUG LOGS ---
|
157 |
+
|
158 |
+
pil_images = []
|
159 |
+
image_basenames_for_prompt = []
|
160 |
+
|
161 |
+
for i, file_data_obj in enumerate(image_inputs): # file_data_obj is one dict from the list
|
162 |
+
img_to_open_path = None
|
163 |
+
current_image_name = file_data_obj.get('name', f"image_{i}.png")
|
164 |
+
|
165 |
+
print(f"Python DEBUG: Processing item {i}: {file_data_obj}, current_image_name: {current_image_name}")
|
166 |
+
|
167 |
+
# For URLs (which is now always the case from Node.js),
|
168 |
+
# Gradio should have downloaded the image and put its local path in file_data_obj.get('path')
|
169 |
+
img_to_open_path = file_data_obj.get('path')
|
170 |
+
if not img_to_open_path:
|
171 |
+
print(f"Error: 'path' was missing in item {i}: {file_data_obj}. Skipping.")
|
172 |
+
continue
|
173 |
+
print(f"Python INFO: Using Gradio-provided path for '{current_image_name}': {img_to_open_path}")
|
174 |
+
|
175 |
+
# Now, process the image using img_to_open_path
|
176 |
+
try:
|
177 |
+
print(f"Python INFO: Opening image from path: {img_to_open_path} (intended name for prompt: {current_image_name})")
|
178 |
+
img = Image.open(img_to_open_path)
|
179 |
+
|
180 |
+
image_basenames_for_prompt.append(os.path.splitext(current_image_name)[0] or f"image_{i}")
|
181 |
+
|
182 |
+
if img.mode == 'RGBA' or img.mode == 'P':
|
183 |
+
print(f"Converting image '{current_image_name}' from {img.mode} to RGB")
|
184 |
+
img = img.convert('RGB')
|
185 |
+
|
186 |
+
processed_img = pipeline.preprocess_image(img)
|
187 |
+
pil_images.append(processed_img)
|
188 |
+
print(f"Image '{current_image_name}' (item {i+1}) processed successfully and added to list.")
|
189 |
+
|
190 |
+
except Exception as e_img_proc:
|
191 |
+
print(f"Error opening or processing image at '{img_to_open_path}' (item {i}, name: '{current_image_name}'): {e_img_proc}")
|
192 |
+
import traceback
|
193 |
+
traceback.print_exc()
|
194 |
+
# Continue to next image if one fails
|
195 |
+
|
196 |
+
# No finally block needed here anymore for deleting temp base64 files
|
197 |
+
|
198 |
+
if not pil_images:
|
199 |
+
print("Error: No images could be processed from the input. Aborting generation.")
|
200 |
+
raise gr.Error("Failed to process any input images.")
|
201 |
+
|
202 |
+
print(f"Python INFO: Total images processed for pipeline: {len(pil_images)}")
|
203 |
+
effective_model_description = model_description
|
204 |
+
if not effective_model_description and image_basenames_for_prompt:
|
205 |
+
effective_model_description = "_prompted_by_" + "_and_".join(image_basenames_for_prompt)
|
206 |
+
effective_model_description = effective_model_description[:100] # Keep it reasonably short
|
207 |
+
elif not effective_model_description:
|
208 |
+
effective_model_description = "ImageGenModel"
|
209 |
+
print(f"Python INFO: Using model_description for upload: {effective_model_description}")
|
210 |
+
|
211 |
+
# Generate 3D model using the Trellis image pipeline
|
212 |
+
try:
|
213 |
+
print(f"Python INFO: Calling internal image_to_3d with {len(pil_images)} images.")
|
214 |
+
# The image_to_3d function expects a list of tuples (PIL.Image, str_filename_or_label)
|
215 |
+
# We have processed_img in pil_images which are already PIL.Image objects after pipeline.preprocess_image
|
216 |
+
# We can use the current_image_name (or derived basenames) as the string part if needed by image_to_3d,
|
217 |
+
# but Trellis's run_multi_image takes a list of PIL images directly.
|
218 |
+
# Let's adapt multiimages for image_to_3d to be List[Tuple[Image.Image, str]]
|
219 |
+
multiimages_for_pipeline = []
|
220 |
+
for idx, p_img in enumerate(pil_images):
|
221 |
+
# Create a simple label for each image for the tuple structure
|
222 |
+
label = image_basenames_for_prompt[idx] if idx < len(image_basenames_for_prompt) else f"image_{idx}"
|
223 |
+
multiimages_for_pipeline.append((p_img, label)) # p_img here is already the *processed* image tensor.
|
224 |
+
# image_to_3d will take image[0] from this list.
|
225 |
+
# This might need adjustment if image_to_3d expects raw PIL Images.
|
226 |
+
# Re-checking Trellis: pipeline.run_multi_image takes List[Image.Image]
|
227 |
+
# and preprocesses them internally if preprocess_image=True (default).
|
228 |
+
# Since we pre-process above, we should pass preprocess_image=False if run_multi_image allows.
|
229 |
+
# The `image_to_3d` in this file is a wrapper for run_multi_image.
|
230 |
+
# It passes `preprocess_image=False`.
|
231 |
+
# So, `pil_images` containing already processed images is what `image_to_3d` expects for `[image[0] for image in multiimages]`
|
232 |
+
|
233 |
+
state, _ = image_to_3d(
|
234 |
+
multiimages=[(img, name) for img, name in zip(pil_images, image_basenames_for_prompt)], # Pass list of (processed_PIL_image, name_str)
|
235 |
+
seed=seed_val,
|
236 |
+
ss_guidance_strength=ss_guidance_strength_val,
|
237 |
+
ss_sampling_steps=ss_sampling_steps_val,
|
238 |
+
slat_guidance_strength=slat_guidance_strength_val,
|
239 |
+
slat_sampling_steps=slat_sampling_steps_val,
|
240 |
+
multiimage_algo=multiimage_algo_val,
|
241 |
+
req=req
|
242 |
+
)
|
243 |
+
if state is None:
|
244 |
+
print("Error: Internal image_to_3d returned None state!")
|
245 |
+
raise ValueError("Internal image_to_3d failed to return state")
|
246 |
+
print(f"Python INFO: Internal image_to_3d completed. State type: {type(state)}")
|
247 |
+
|
248 |
+
print("Python INFO: Calling internal extract_glb...")
|
249 |
+
glb_path, _ = extract_glb(
|
250 |
+
state, mesh_simplify_val, texture_size_val, req
|
251 |
+
)
|
252 |
+
if glb_path is None or not os.path.isfile(glb_path):
|
253 |
+
print(f"Error: Internal extract_glb returned None or invalid path: {glb_path}")
|
254 |
+
raise FileNotFoundError(f"Generated GLB file not found at {glb_path}")
|
255 |
+
print(f"Python INFO: Internal extract_glb completed. GLB path: {glb_path}")
|
256 |
+
|
257 |
+
print(f"Python INFO: Uploading GLB from {glb_path} to {NODE_SERVER_UPLOAD_URL}")
|
258 |
+
persistent_url = None
|
259 |
+
with open(glb_path, "rb") as f:
|
260 |
+
files = {"modelFile": (os.path.basename(glb_path), f, "model/gltf-binary")}
|
261 |
+
payload = {
|
262 |
+
"clientType": "imagen",
|
263 |
+
"modelStage": "imagen_mesh",
|
264 |
+
"prompt": effective_model_description # Use the description here
|
265 |
+
}
|
266 |
+
print(f"Python INFO: Upload payload: {payload}")
|
267 |
+
response = requests.post(NODE_SERVER_UPLOAD_URL, files=files, data=payload)
|
268 |
+
response.raise_for_status() # Raise an exception for bad status codes
|
269 |
+
result = response.json()
|
270 |
+
persistent_url = result.get("persistentUrl")
|
271 |
+
if not persistent_url:
|
272 |
+
print(f"Error: No persistent URL in Node.js server response: {result}")
|
273 |
+
raise ValueError("Upload successful, but no persistent URL returned")
|
274 |
+
print(f"Python INFO: Successfully uploaded to Node server. Persistent URL: {persistent_url}")
|
275 |
+
return persistent_url
|
276 |
+
|
277 |
+
except Exception as e:
|
278 |
+
print(f"ERROR in Image-to-3D pipeline: {e}")
|
279 |
+
import traceback
|
280 |
+
traceback.print_exc()
|
281 |
+
raise gr.Error(f"Image-to-3D pipeline failed: {e}")
|
282 |
+
|
283 |
+
# Interfaz Gradio
|
284 |
+
with gr.Blocks(delete_cache=(600, 600)) as demo:
|
285 |
+
gr.Markdown("""
|
286 |
+
# UTPL - Conversión de Multiples Imágenes a objetos 3D usando IA
|
287 |
+
### Tesis: *"Objetos tridimensionales creados por IA: Innovación en entornos virtuales"*
|
288 |
+
**Autor:** Carlos Vargas
|
289 |
+
**Base técnica:** Adaptación de [TRELLIS](https://trellis3d.github.io/) (herramienta de código abierto para generación 3D)
|
290 |
+
**Propósito educativo:** Demostraciones académicas e Investigación en modelado 3D automático
|
291 |
+
""")
|
292 |
+
|
293 |
+
with gr.Row():
|
294 |
+
with gr.Column():
|
295 |
+
with gr.Tabs() as input_tabs:
|
296 |
+
with gr.Tab(label="Multiple Images", id=1) as multiimage_input_tab:
|
297 |
+
multiimage_prompt = gr.Gallery(label="Image Prompt", format="png", type="pil", height=300, columns=3)
|
298 |
+
|
299 |
+
with gr.Accordion(label="Generation Settings", open=False):
|
300 |
+
seed = gr.Slider(0, MAX_SEED, label="Seed", value=0, step=1)
|
301 |
+
randomize_seed = gr.Checkbox(label="Randomize Seed", value=True)
|
302 |
+
gr.Markdown("Stage 1: Sparse Structure Generation")
|
303 |
+
with gr.Row():
|
304 |
+
ss_guidance_strength = gr.Slider(0.0, 10.0, label="Guidance Strength", value=7.5, step=0.1)
|
305 |
+
ss_sampling_steps = gr.Slider(1, 50, label="Sampling Steps", value=12, step=1)
|
306 |
+
gr.Markdown("Stage 2: Structured Latent Generation")
|
307 |
+
with gr.Row():
|
308 |
+
slat_guidance_strength = gr.Slider(0.0, 10.0, label="Guidance Strength", value=3.0, step=0.1)
|
309 |
+
slat_sampling_steps = gr.Slider(1, 50, label="Sampling Steps", value=12, step=1)
|
310 |
+
multiimage_algo = gr.Radio(["stochastic", "multidiffusion"], label="Multi-image Algorithm", value="stochastic")
|
311 |
+
|
312 |
+
generate_btn = gr.Button("Generate")
|
313 |
+
|
314 |
+
with gr.Accordion(label="GLB Extraction Settings", open=False):
|
315 |
+
mesh_simplify = gr.Slider(0.9, 0.98, label="Simplify", value=0.95, step=0.01)
|
316 |
+
texture_size = gr.Slider(512, 2048, label="Texture Size", value=1024, step=512)
|
317 |
+
|
318 |
+
extract_glb_btn = gr.Button("Extract GLB", interactive=False)
|
319 |
+
|
320 |
+
with gr.Column():
|
321 |
+
video_output = gr.Video(label="Generated 3D Asset", autoplay=True, loop=True, height=300)
|
322 |
+
model_output = gr.Model3D(label="Extracted GLB", height=300)
|
323 |
+
download_glb = gr.DownloadButton(label="Download GLB", interactive=False)
|
324 |
+
|
325 |
+
output_buf = gr.State()
|
326 |
+
|
327 |
+
# Manejadores
|
328 |
+
demo.load(start_session)
|
329 |
+
demo.unload(end_session)
|
330 |
+
|
331 |
+
multiimage_prompt.upload(
|
332 |
+
preprocess_images,
|
333 |
+
inputs=[multiimage_prompt],
|
334 |
+
outputs=[multiimage_prompt],
|
335 |
+
)
|
336 |
+
|
337 |
+
generate_btn.click(
|
338 |
+
get_seed,
|
339 |
+
inputs=[randomize_seed, seed],
|
340 |
+
outputs=[seed],
|
341 |
+
).then(
|
342 |
+
image_to_3d,
|
343 |
+
inputs=[multiimage_prompt, seed, ss_guidance_strength, ss_sampling_steps, slat_guidance_strength, slat_sampling_steps, multiimage_algo],
|
344 |
+
outputs=[output_buf, video_output],
|
345 |
+
).then(
|
346 |
+
lambda: gr.Button(interactive=True),
|
347 |
+
outputs=[extract_glb_btn],
|
348 |
+
)
|
349 |
+
|
350 |
+
video_output.clear(
|
351 |
+
lambda: gr.Button(interactive=False),
|
352 |
+
outputs=[extract_glb_btn],
|
353 |
+
)
|
354 |
+
|
355 |
+
extract_glb_btn.click(
|
356 |
+
extract_glb,
|
357 |
+
inputs=[output_buf, mesh_simplify, texture_size],
|
358 |
+
outputs=[model_output, download_glb],
|
359 |
+
).then(
|
360 |
+
lambda: gr.Button(interactive=True),
|
361 |
+
outputs=[download_glb],
|
362 |
+
)
|
363 |
+
|
364 |
+
model_output.clear(
|
365 |
+
lambda: gr.Button(interactive=False),
|
366 |
+
outputs=[download_glb],
|
367 |
+
)
|
368 |
+
|
369 |
+
# --- Add this section to explicitly register the API function for image to 3D ---
|
370 |
+
# Use gr.JSON for the API input, as it's designed for arbitrary JSON data.
|
371 |
+
api_image_inputs_json = gr.JSON(value=[]) # <--- CHANGED to gr.JSON and new variable name
|
372 |
+
api_input_type_state = gr.State(value="url")
|
373 |
+
api_model_description_state = gr.State(value="ImagenModel")
|
374 |
+
|
375 |
+
with gr.Row(visible=False): # Hide this row in the UI
|
376 |
+
api_image_gen_trigger_btn = gr.Button("API Image-to-3D Trigger")
|
377 |
+
|
378 |
+
api_image_gen_output_url = gr.Textbox(label="Generated Model URL (API)", visible=False)
|
379 |
+
|
380 |
+
api_image_gen_trigger_btn.click(
|
381 |
+
generate_model_from_images_and_upload,
|
382 |
+
inputs=[
|
383 |
+
api_image_inputs_json, # <--- CHANGED to use the gr.JSON component's variable name
|
384 |
+
api_input_type_state,
|
385 |
+
seed,
|
386 |
+
ss_guidance_strength,
|
387 |
+
ss_sampling_steps,
|
388 |
+
slat_guidance_strength,
|
389 |
+
slat_sampling_steps,
|
390 |
+
multiimage_algo,
|
391 |
+
mesh_simplify,
|
392 |
+
texture_size,
|
393 |
+
api_model_description_state,
|
394 |
+
],
|
395 |
+
outputs=[api_image_gen_output_url],
|
396 |
+
api_name="generate_model_from_images_and_upload"
|
397 |
+
)
|
398 |
+
# --- End API registration section ---
|
399 |
+
|
400 |
+
# Lanzar la aplicación Gradio
|
401 |
+
if __name__ == "__main__":
|
402 |
+
pipeline = TrellisImageTo3DPipeline.from_pretrained("cavargas10/TRELLIS")
|
403 |
+
pipeline.cuda()
|
404 |
+
try:
|
405 |
+
pipeline.preprocess_image(Image.fromarray(np.zeros((512, 512, 3), dtype=np.uint8))) # Precargar rembg
|
406 |
+
except:
|
407 |
+
pass
|
408 |
+
demo.launch(show_error=True)
|
assets/example_image/T.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_building_building.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_building_castle.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_building_colorful_cottage.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_building_maya_pyramid.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_building_mushroom.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_building_space_station.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_creature_dragon.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_creature_elephant.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_creature_furry.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_creature_quadruped.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_creature_robot_crab.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_creature_robot_dinosour.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_creature_rock_monster.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_humanoid_block_robot.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_humanoid_dragonborn.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_humanoid_dwarf.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_humanoid_goblin.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_humanoid_mech.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_crate.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_fireplace.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_gate.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_lantern.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_magicbook.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_mailbox.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_monster_chest.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_paper_machine.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_phonograph.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_portal2.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_storage_chest.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_telephone.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_television.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_misc_workbench.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_vehicle_biplane.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_vehicle_bulldozer.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_vehicle_cart.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_vehicle_excavator.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_vehicle_helicopter.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_vehicle_locomotive.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/typical_vehicle_pirate_ship.png
ADDED
![]() |
Git LFS Details
|
assets/example_image/weatherworn_misc_paper_machine3.png
ADDED
![]() |
Git LFS Details
|
assets/example_multi_image/character_1.png
ADDED
![]() |
Git LFS Details
|
assets/example_multi_image/character_2.png
ADDED
![]() |
Git LFS Details
|
assets/example_multi_image/character_3.png
ADDED
![]() |
Git LFS Details
|
assets/example_multi_image/mushroom_1.png
ADDED
![]() |
Git LFS Details
|
assets/example_multi_image/mushroom_2.png
ADDED
![]() |
Git LFS Details
|
assets/example_multi_image/mushroom_3.png
ADDED
![]() |
Git LFS Details
|