Gen AI

3D Scene Representation

Instead of storing a 3D video game level using millions of tiny triangles (polygons), NeRFs memorize the entire 3D room inside the weights of a neural network, calculating what the room looks like on-the-fly based on where you are standing.

NeRFs use neural networks to predict the color and density of a scene from any angle, effectively storing a fully explorable 3D world inside the weights of a model.

Why Does This Exist?

In traditional computer graphics (like video games or Pixar movies), 3D objects are created using meshes—hollow shells made of millions of tiny connected triangles (polygons). To render an image, the computer calculates how light bounces off every single triangle. This is mathematically exact, but it is extremely hard to generate from scratch. If you take 50 photos of a real-world statue and try to automatically build a perfect polygon mesh from those photos, the result is usually lumpy, jagged, and missing textures.

Neural Radiance Fields (NeRFs) solved the "Novel View Synthesis" problem using a radically different approach. Instead of trying to build a physical 3D model, what if we just trained a neural network to memorize how the light looks? If you give the neural network an X,Y,Z coordinate in the room, it will simply output what color is at that coordinate. The entire 3D scene is not stored as triangles; it is stored purely as knowledge inside the weights of a neural network.

Think of It Like This

The Omniscient Surveyor

Imagine you want to recreate a detailed sculpture of a dragon.

Traditional 3D (Photogrammetry): You try to sculpt a replica out of clay by looking at photographs. It's difficult to get the exact curves and shadows right.
NeRF: You lock a very smart surveyor in an empty room. You give them a laser pointer and 50 photographs of the dragon from different angles. You say, "If I stand exactly here and point my laser exactly there, what color will the laser hit?" The surveyor studies the photos until they can perfectly answer that question for any angle in the room.

The surveyor (the neural network) has perfectly memorized the 3D space, even though they haven't sculpted a single piece of clay.

How It Actually Works

NeRFs treat the 3D world as a continuous mathematical function rather than a grid of discrete objects.

1. The Inputs and Outputs

The neural network inside a NeRF (usually a simple Multi-Layer Perceptron, not a CNN) takes in exactly 5 numbers:

The 3D coordinates in space: $x, y, z$
The viewing angle (where the camera is looking from): $\theta, \phi$

It outputs exactly 2 things:

The RGB color at that exact point.
The Density ( $\sigma$ ) at that exact point (how transparent or solid the point is).

2. Ray Marching (Volumetric Rendering)

To render a 2D image of the scene, the computer simulates a camera. For every single pixel on your computer screen, it shoots a mathematical "ray" out into the 3D space. As the ray travels through the space, it samples the neural network hundreds of times. "Network, what is the color and density here? How about an inch further? How about here?" The ray accumulates the color based on the density. If it hits empty space (density = 0), it keeps going. If it hits the surface of a red apple (density = 1, color = red), the ray stops, and that pixel on your screen is colored red.

3. Training the NeRF

To train the NeRF, you give it 50 photos of a scene (along with the exact camera angles the photos were taken from). You shoot rays from the virtual camera, ask the network to predict the pixel colors, and compare the prediction to the actual photograph. If the network is wrong, you use backpropagation to adjust the weights. Slowly, the network learns to carve out a perfect 3D representation of the scene to minimize the error across all 50 photographs.

4. 3D Gaussian Splatting (The Modern Evolution)

While NeRFs produce beautiful results, shooting millions of rays and querying a neural network hundreds of times per ray is incredibly slow (it can take a minute to render a single frame). In 2023, 3D Gaussian Splatting revolutionized the field. Instead of querying a neural network with a ray, it represents the scene using millions of tiny, fuzzy 3D blobs (Gaussians). It projects these blobs directly onto the 2D screen mathematically. This bypassed the neural network entirely for rendering, allowing photorealistic 3D scenes to be rendered at 120 Frames Per Second in real-time.

Show Me the Code

This pseudocode shows the core concept of Ray Marching: querying the network along a ray to figure out the final color of a single pixel.

1import torch2
3def render_ray(nerf_model, ray_origin, ray_direction, num_samples=64, step_size=0.1):4    """5    Shoots a single ray through the scene and calculates the final pixel color.6    """7    accumulated_color = torch.zeros(3) # RGB (0,0,0)8    transmittance = 1.0 # Starts fully transparent (100% light passes through)9    10    # March along the ray in small steps11    for i in range(num_samples):12        # Calculate the exact X,Y,Z coordinate at this step13        current_position = ray_origin + (ray_direction * i * step_size)14        15        # Ask the neural network for the color and density at this exact spot16        # (Assuming the viewing angle is also passed into the model)17        predicted_color, density = nerf_model(current_position, ray_direction)18        19        # Calculate how much light is blocked by the density at this point20        alpha = 1.0 - torch.exp(-density * step_size)21        22        # The contribution to the final pixel color is the color of this point,23        # multiplied by its solidness (alpha), multiplied by how much light 24        # actually made it this far (transmittance).25        weight = transmittance * alpha26        accumulated_color += weight * predicted_color27        28        # Update transmittance (if we hit something solid, transmittance drops to 0)29        transmittance *= (1.0 - alpha)30        31        # Early stopping: If the ray hits a completely solid wall, stop calculating32        if transmittance < 0.01:33            break34            35    return accumulated_color

Watch Out For

The Baked Lighting Problem

Because a standard NeRF memorizes the scene exactly as it appeared in the 50 photographs, the lighting and shadows are permanently "baked" into the neural network. If you render the scene, you cannot easily change the position of the sun or add a new light source, because the NeRF doesn't know what a "light source" is; it just knows that a specific coordinate happens to be bright yellow. Advanced extensions (like Relightable NeRFs) are required to separate the 3D geometry from the lighting conditions.

The Quick Version

Traditional computer graphics use complex polygon meshes to represent 3D scenes.
Neural Radiance Fields (NeRFs) use a neural network to memorize the scene. The network takes in an X,Y,Z coordinate and outputs the Color and Density at that point.
To generate an image, the system shoots virtual "rays" from the camera, querying the neural network hundreds of times along the ray to determine the final pixel color (Ray Marching).
NeRFs can perfectly recreate photorealistic, reflective 3D scenes from just a handful of 2D photographs.
3D Gaussian Splatting is a modern alternative that replaces the slow neural network queries with millions of tiny 3D blobs, enabling real-time rendering.