GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Rendering

Kai Ye1*, Chong Gao2*, Guanbin Li2, Wenzheng Chen1†, Baoquan Chen1†
1Peking University, 2Sun Yat-sen University
(*equal contribution, †equal advisory)
Paper arXiv Code (Comming Soon)

Inverse Rendering (in 15 minutes): (a) PBR Color; (b) Albedo; (c) Mesh; (d) Environment Light.

Abstract

We consider the problem of physically-based inverse rendering using 3D Gaussian Splatting (3DGS) representations Kerbl et al. (2023b). While recent 3DGS methods have achieved remarkable results in novel view synthesis (NVS), accurately capturing high-fidelity geometry, physically interpretable materials and lighting remains challenging, as it requires precise geometry modeling to provide accurate surface normals, along with physically-based rendering (PBR) techniques to ensure correct material and lighting disentanglement.

Previous 3DGS methods resort to approximating surface normals, but often struggle with noisy local geometry, leading to inaccurate normal estimation and suboptimal material-lighting decomposition. In this paper, we introduce GeoSplatting, a novel hybrid representation that augments 3DGS with explicit geometric guidance and differentiable PBR equations. Specifically, we bridge isosurface and 3DGS together, where we first extract isosurface mesh from a scalar field, then convert it into 3DGS points and formulate PBR equations for them in a fully differentiable manner.

In GeoSplatting, 3DGS is grounded on the mesh geometry, enabling precise surface normal modeling, which facilitates the use of PBR frameworks for material decomposition. This approach further maintains the efficiency and quality of NVS from 3DGS while ensuring accurate geometry from the isosurface. Comprehensive evaluations across diverse datasets demonstrate the superiority of GeoSplatting, consistently outperforming existing methods both quantitatively and qualitatively.

Overview

Overview

Pipeline. GeoSplatting first extracts an intermediate mesh from the scalar field, upon which Gaussian points are sampled and rendered using PBR equations. Finally, they are composited into images through the Gaussian rasterization pipeline. The entire process is fully differentiable.

Novel View Synthesis

Novel View Synthesis

Qualitative NVS comparison on NeRF dataset. Our method effectively recovers complex geometries, detailed textures, and non-Lambertian appearances, as shown in the sub-windows

Geometry Recovery

Geomtry Recovery

Qualitative Geometry Comparison. Our method achieves accurate geometry in scenes with challenging lighting and material conditions (shiny Spot) and complex topology (Damicornis).

Decomposition

Decomposition

PBR Rendering Decomposition. Our PBR framework successfully disentangles the materials and lighting, capturing meaningful diffuse, specular and residual terms. Note the residual image even learns the inter-reflection effects (the most shiny ball reflects a small green ball).

Relighting Demo

Video Demo. Click the button to enable visualization.

Realistic Dataset Results

Realistic Dataset Realistic Dataset

BibTeX

@misc{ye2024geosplattinggeometryguidedgaussian,
    title={GeoSplatting: Towards Geometry Guided Gaussian Splatting for Physically-based Inverse Rendering}, 
    author={Kai Ye and Chong Gao and Guanbin Li and Wenzheng Chen and Baoquan Chen},
    year={2024},
    eprint={2410.24204},
    archivePrefix={arXiv},
    primaryClass={cs.CV},
    url={https://arxiv.org/abs/2410.24204}, 
}