Mixed Reality Diffusion prototype as a tool for exploring concepts, styles, and moods by transforming real-world surroundings into alternate realities. #MixedReality #MR #AI #StableDiffusion #Quest3

We turned dreams into worlds. Then filled them with history's greatest minds. Not a video. A world, running directly in your browser. Step inside ↓

A quick demo of Manifold, a small control surface I just built for my realtime AI pipeline. Like a patchbay, it routes the output of one tool into the next: Hands-on Hallucination generates frames from a live camera feed, Embiggen upscales/restores them, and Unflatten turns them into 3D Gaussian Splat assets. Manifold wires those stages together, lets me start/stop each tool depending on GPU memory, and gives me one place to monitor the whole chain, including previewing the final 3DGS directly in the interface. The loop is becoming much easier to control. Running on a 4090, I still need to split the chain in two (Hands-on Hallucination Embiggen first, then Unflatten) because of VRAM limits. Tomorrow, on a 5090, I should be able to have all three tools running in parallel. #diffusion #3DGS #TripoSplat

3DGS through the new TripoSplat ← upscale ← realtime img2img ← camera ← physical interface. Same video, but reversed, as the first one may have been a bit long. This one starts with the generated 3DGS fragments before going back to the magic clay interface and real-time img2img pipeline that are at the origin of them. Next: - better input and camera's pov, - splitting it across two machines to automate the hand-off, - and adding in the Tripo3D API for higher-res models... the 3D printer is just waiting for that. #diffusion #3DGS #TripoSplat

A quick first pass at a new iteration of the realtime diffusion pipeline: physical interface → camera → realtime img2img → upscale → 3DGS through the new TripoSplat. Rough and held together with tape for now, but the whole loop runs end to end. Next: - better input and camera's pov, - splitting it across two machines to automate the hand-off, - and adding in the Tripo3D API for higher-res models... the 3D printer is just waiting for that. #diffusion #3DGS #TripoSplat

Back to capturing my neighborhood... this time, our back alley again. Here is just a quick initial exploration and reconstruction check of this capture. Captured with the PortalCam in between two rain showers. #GaussianSplatting #3DGS #pointcloud #portalcam

Back to capturing my neighborhood... this time, Coiffure Mile-End played the game. Here is just a quick initial exploration and reconstruction check of this capture. The mirrors facing each other made for an interesting challenge, along with the constantly changing light as the sun kept appearing and disappearing behind clouds. #GaussianSplatting #3DGS #pointcloud #portalcam

The scenes coming out of The Latent Space (a collaboration with Alejandro Cartagena / @halecar2 ) are now becoming explorable world fragments. Using the LoRAs trained on Alejandro's datasets, paper compositions on my desk are passed through the realtime Diffusion pipeline, then turned into 3D Gaussian Splats using ML Sharp.⁣ ⁣ #diffsuion #GaussianSplatting #MLSharp #SparkJS #AI

A sample of the world fragments from yesterday's Hands-on Hallucination session. Physical interfaces → real-time img2img diffusion → 3D GS → real-time rendering in a browser on desktop, mobile and AVP.⁣ #diffsuion #GaussianSplatting #MLSharp #SparkJS #AI

Accelerated capture from yesterday's Hands-on Hallucination session. I started out testing the system alone, then my son joined in and quickly took control; deciding which generated frames to freeze and transform into 3D Gaussian Splats. We ended up with a series of strange little world fragments.⁣⁣ ⁣ Physical interfaces → real-time img2img diffusion → 3D GS → real-time rendering in a browser on desktop, mobile and AVP.⁣ ⁣⁣ #GaussianSplatting #AppleML #MLSharp #SparkJS

Really happy to see Dormant Memories featured as part of @theworldlabs' Community Showcase. worldlabs.ai/labs/showcase/d… Dormant Memories is an ongoing exploration around volumetric capture, 3D Gaussian Splats, real-time rendering, and world manipulation. #GaussianSplatting #WorldLabs #RealtimeGraphics #AI #3DGS

I resumed work on my Hands-on Hallucination ML Sharp experiments from ~5 months ago. I manipulate paper, clay, etc. in front of a camera, watch a real-time diffusion pipeline hallucinate a world from them, press a knob, and the frozen frame becomes a 3D Gaussian Splat, instantly explorable in the browser or in VR. #GaussianSplatting #AppleML #MLSharp #SparkJS