The pipeline of our proposed method: As seen in the left figure, given an input image and a target body, we employ body-conditioned image generation to produce body-aligned consistent four-view orthographic images. The four-view images are then fed into a native 3D diffusion model to obtain the asset shape.The similarity transformation (Sim3) of the generated asset is estimated through silhouette-based projection optimization. Finally, after solving the body-asset penetration, the Sim3-transformed asset is fitted onto the human body. As seen in the right figure, there are four methods to acqure input body and image pairs. (a) SMPLX Fitting. (b)Sketch-Based Modeling. (c) Virtual Try-on. (d) Manual Images Assembly.

Sim(3) Estimation: Due to the non-deterministic nature of the diffusion model, the shape generator cannot guarantee that the output shape will align with the input multi-view images. This misalignment can manifest as differences in scale, translation, and rotation. To address this issue, we apply the Similarity Transformation Estimation.

Asset-Body Penetration Handling: The Sim(3)-transformed asset can roughly align with the body, but small penetrations between the asset and body still exist. To address this issue, we further resolve such penetrations using a physics simulation-based solver.

Result gallery across four settings: (a) Single-view reconstruction with fitted SMPLX; (b) Image-Based Virtual Try-On; (c) Assembling existing 2D assets; and (d) Sketch-Based modeling. Each image is followed by the reconstructed asset mesh. Our method demonstrates proficiency in effectively producing body-aligned asset shapes and faithfully capturing high-fidelity geometric details from the input.

As observed, BCNet, ClothWild and SewFormer fail to generate results that align with the input image. This limitation stems from their sole reliance on garment parametric models (whether explicit, implicit, or based on sewing patterns). The limited representational capacity of these parametric models hinders their ability to recover aligned garment shapes that accurately follows the input images. Although Frankenstein employs a more effective 3D triplane representation, its stringent requirements for training data make it difficult to scale up. The limited availability of data results in inadequate generalization capabilities of the trained model, hindering its ability to effectively reconstruct complex garments from images. By harnessing the advanced generalization capabilities of large 3D models, Garment3DGen can reconstruct relatively aligned 3D shapes from images. However, due to the lack of prior knowledge about body shapes, the generated 3D garments are challenging to fit onto the human body, which necessitates labor-intensive and time-consuming manual post-processing to achieve body-garment alignment. Furthermore, its template-fitting method restricts topological flexibility, obstructing the recovery of complex structures and fine-grained geometric details. Our method demonstrates proficiency in effectively producing body-aligned garment shapes and faithfully capturing high-fidelity geometric details from single-view images.

As observed, both the single-view and body-agnostic multi-view approaches result in body-garment misalignment. The lack of depth information in the single-view to 3D asset method often leads to discrepancies in depth between the body and the asset. The absence of body conditioning in the original multi-view generation yields multi-view images that misalign with the corresponding 2D body renderings, causing noticeable offsets between the generated multi-view images and the body renderings. In contrast, our body-aligned multi-view image generation guarantees precise alignment of assets with the human body across multiple perspectives, ensuring that the final generated 3D assets are consistently aligned with the 3D body.

Qualitative comparison between ours and the ablated alignement strategies. As shown, while the generated multi-view images align with the rendered body from various 2D perspectives (b), the generated 3D assets do not effectively guarantee alignment with the input images (c). This discrepancy stems from different techniques and normalization strategies used in multiview-to-3D diffusion models. By employing $\Sim(3)$ optimization, the 3D mesh aligns with the human body (d), though some penetrations remain. Our penetration-handling approach, as illustrated in (e), achieves a penetration-free state for the asset and the body.