One possibility is to consider each object in the scene, determine what pixels the object maps to and assign the proper pixel colors using the reflectance model calculations.
The difficulty with this approach is there are (theoretically) an infinite number of points on each object, and therefore many more than one point on each object maps to the same pixel.