Gaussian splatting + WebGL

[brentyi]

I spent some time studying + adapting https://github.com/antimatter15/splat/. It's really cool stuff.

splat.mp4

Work in progress: there's a lot of low hanging fruit for optimization, and the current implementation clashes with the <EffectComposer /> we use for outline/hover effects.

[brgrp]

Hey, I just tried to run the demo with Chrome and Firefox, but my display is getting incredibly slow. GPU and CPU usage stays low.

Ubuntu 22.04 LTS + GTX3080
Chrome: 117.0.5938.132 (64-bit)
Firefox: 117.0.1 (64-bit)
NVIDIA-SMI 525.125.06 Driver Version: 525.125.06

Does anyone have an idea what the problem could be?

[jkulhanek]

Can you confirm hardware acceleration is enabled (chrome://gpu)?

[brgrp]

Can you confirm hardware acceleration is enabled (chrome://gpu)?

Graphics Feature Status

• Canvas: Hardware accelerated
• Canvas out-of-process rasterization: Enabled
• Direct Rendering Display Compositor: Disabled
• Compositing: Hardware accelerated
• Multiple Raster Threads: Enabled
• OpenGL: Enabled
• Rasterization: Hardware accelerated
• Raw Draw: Disabled
• Skia Graphite: Disabled
• Video Decode: Hardware accelerated
• Video Encode: Software only. Hardware acceleration disabled
• Vulkan: Disabled
• WebGL: Hardware accelerated
• WebGL2: Hardware accelerated
• WebGPU: Hardware accelerated

[brentyi]

Which splat file are you loading? Splats are currently rendering smoothly in Chrome on both my laptop (M1 Mac) and phone (Samsung S21+).

[brentyi]

sort timing notes:

# Gaussians	270K	1M
Array.sort	79ms	380ms
JS counting sort	25ms	140ms
C++ counting sort	14ms	53ms

[jkulhanek]

What is the current state of things? Is it ready to be merged soon?

[brentyi]

I'm leaning toward closing the PR; it's working, but:

• Still clashes with outline effects / EffectComposer.
• Doesn't sort properly when there are multiple splat files.
• Can be optimized a lot. (in particular I would want to move more data into textures, as opposed to attributes)

After I implemented this splatting capabilities have also popped up in external libraries (example: https://github.com/pmndrs/drei#Splat); it may make sense to adapt one of these instead.