Generating realistic human avatars in complex motions—such as clothing dynamics—requires modeling of global and local deformations which remains challenging in monocular settings. We address this problem by leveraging neural Jacobian fields (NJFs) for representing semi-rigid deformations. We train self-supervised neural networks for predicting Jacobian matrices that give the pose-dependent deformations, by solving a Poisson equation. However, monocular input presents several difficulties such as self-occluded regions and invisible surfaces. To address these issues, we introduce three key components: a constrained Poisson solver, signed distance-based Jacobian regularization, and a deformation-guided residual flow loss, which together suppress boundary artifacts, recover frequently occluded regions such as armpits and thighs, and enforce temporal consistency during motion. Experiments on benchmark and in-the-wild videos demonstrate that our method generates temporally stable and geometrically coherent avatars, outperforming state-of-the-art approaches.
A geometry branch and an appearance branch. From the initial mesh, an MLP predicts a Jacobian for every face, a screened Poisson solver turns it into vertex displacements (coarse first, then fine), LBS poses the mesh, and 3D Gaussians on the faces render the avatar.
Coarse-to-fine Jacobian fields. A coarse field on the canonical mesh $\mathcal{M}^c$ captures large deformations; a fine field on the twice-subdivided mesh $\mathcal{M}^f$ adds wrinkles, taking the coarse feature $\mathbf{z}^c$ as extra input. Both are summed before LBS.
Refining the MHR template with a differentiable renderer based on previous work*, supervised by Sapiens normal maps and segmentation masks, lets the pipeline initially reconstruct avatars even in loose clothing. Heatmaps show the geometric updates.
* Large Steps in Inverse Rendering of Geometry (Nicolet et al., SIGGRAPH Asia 2021).
In neural Jacobian fields (NJF), the Jacobian $J_i$ is the gradient of the deformation map on each face: instead of predicting where the vertices go, the network predicts how every triangle stretches, rotates, and shears. Given the Jacobian field, the deformation map $\Phi^*$ is recovered by solving a Poisson equation.
Occluded and never-observed surfaces get no constraint in the standard Poisson solver, so the mesh stretches. A screening term keeps the deformed vertices close to their canonical positions $\mathcal{V}$.
SDF-based regularizer. An SDF voxel grid built from the initial mesh gives normals $\hat{\mathbf{n}}_{\mathrm{SDF}}$; aligning the deformed avatar's normals to them keeps the global shape intact in rarely seen, articulated regions such as the armpits.
Deformation-guided residual flow loss. The LBS motion between frames $t-1$ and $t$ is projected into a displacement map $\mathbf{W}^t$, refined by a pretrained optical-flow network, and only the residual is minimized, which keeps the deformation temporally consistent.
SDF-based regularizer
Deformation-guided residual flow loss
For each scene, we render the mesh of our avatar: the ground-truth image of a test frame (left), the canonical avatar (center), and the posed avatar (right).
Compared methods suffer from texture-copying artifacts on the geometry, whereas our method recovers smooth surfaces in textured regions and sharp wrinkles.
Qualitative comparison of rendering quality with state-of-the-art methods on the MonoPerfCap dataset. Ours shows sharper textures from side views.
@inproceedings{won2026jacobianavatar,
title = {JacobianAvatar: Temporally Consistent Semi-rigid Avatar Reconstruction from a Monocular Video},
author = {Won, Changyeon and Park, Min-Gyu and Park, Seonghwan and Yoon, Ju Hong and Jeon, Hae-Gon},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}