Google Researchers Add Powerful Denoise Tool to NeRF AI Program

moped

Google researchers have unveiled a new artificially intelligent (AI) program that has incredible denoising ability allowing users to “see in the dark.”

RawNeRF is able to combine images taken from many different camera viewpoints to jointly denoise and reconstruct the scene.

It works in tandem with Google’s Neural Radiance Fields,known as NeRF, a view synthesis that creates 3D renderings of a scene from multiple photos.

Ben Mildenhall, a research scientist, gives an example of a table lit only by candlelight that he took on his smartphone. The RAW file was extremely noisy when tried to be brightened.

However, Mildenhall processed it through RawNeRF and the image returned with almost no noise and much more detail.

Mildenhall says that RawNeRF does more than just denoising. This allows us to adjust the camera’s position so we can view the scene at different angles.

“RawNeRF goes beyond this, we actually reconstruct the scene and render new views in a linear hdr color space. This means we can also do things, like vary the exposure, by rescaling renderings in linear space before they’re post-processed.”

moped

RawNeRF can swap out the basic gamma curve for a more complex tone mapping algorithm that brings out detail in shadows.

Mildenhall says that linear colors can be used to create synthetic defocus effects with precise bokeh effects.

Moving on from the candlelit table scene, Mildenhall takes the technology to a photo of a dark road lit only by street lights.

Again, the RawNeRF performs remarkably well and this is because it was trained on RAW image data rather than post-process jpegs.

Mildenhall then shows further examples of badly lit, noisy images that are rescued incredibly well by the RawNeRF.

“Despite the noise, render is able to recover the fine geometry of this fence and bicycle,” he explains over one image.

The software is fed a collection of five to 200 photos of the same scene, which is easily shot in seconds using today’s camera technology.

From the data, the AI software can manipulate focus, exposure, and tone mapping after the fact.

“Although a single raw image appears significantly more noisy than a post-processed one, we show that NeRF is highly robust to the zero-mean distribution of raw noise,” writes the researchers.

“When optimized over many noisy raw inputs (25-200), NeRF produces a scene representation so accurate that its rendered novel views outperform dedicated single and multi-image deep raw denoisers run on the same wide baseline input images,” they add.

“As a result, our method, which we call RawNeRF, can reconstruct scenes from extremely noisy images captured in near-darkness.”

The team’s paper can be read here.


Image credits: All photos by Ben Mildenhall.

Loading...