Very cool work! We followed a quite similar pipeline a couple of year ago to generate scenes from a single-exemplar: arxiv.org/abs/2412.16253. In our case, latent features were DINO and Gaussians were decoded with a patch-based synthesis step (inspired by PatchMatch).
MetroGS: Efficient and Stable Reconstruction of Geometrically Accurate High-Fidelity Large-Scale Scenes
Kehua Chen, et al.
tl;dr: 2DGS dense enhancement with π3 monocular&PatchMatch-based multi-view opt. depth-guided appearance modeling with Tri-MipRF
arxiv.org/abs/2511.19172
I think we needed mature non-learned systems to generate training data for Dust3r and others. A lot of the datasets used for training are generated with SLAM, SfM, PatchMatch MVS, etc.
That is patchmatch based though? You dont get the precision of traditional methods with current learning based, but the fusion step actually can solve most issues.
My original statement still stands though ;)
While being great, this is a license centipede 😅. The repo is MIT, which uses agpl licensed code, which implements ebsynth, which uses an Adobe patented patchmatch Algo. 🥲
PatchMatch was pure magic when it was introduced 14 years ago. It allowed you to shrink images, delete and even move objects around. It eventually turned into Photoshop content-aware fill.
I have very little intuition for what matters in patchmatch actually, but a very big distribution shift between proposals and actual seems like it should matter.
Another thing that I found quite useful in improving patchmatch output is this refinement approach: github.com/rossimattia/depth…. You might give it a shot.
Here's an example, I think you can see some artifacts of patchmatch. For example that you tend to get "constant depth" regions, that can bleed outside the object boundaries.
To be fair with patchmatch, there have been significant improvement over colmap impl (e.g. ieeexplore.ieee.org/document…). In my own tests, it is consistently better. I think a good contribution to the community would be to port the code to colmap in terms of style and UX/UI.