Tips and Tricks about 3D Scene Reconstruction

I am using Computer Vision System toolbox in Matlab 2014 for a project that requires detecting 3D profile of an arbitrary object and coordinates of points on its surface using stereo imaging. The documentation the MathWorks has provided suggests that it could be done using this toolbox; specifically through reconstructScene function.

Well, I am able to calibrate a set of stereo cameras accurately in a code I have developed but when I use the calibration data to reconstruct a 3D scene using a pair of images, it just doesn't look right. The image is very fuzzy and instead of giving me a reconstructed profile of the scene, it more or so looks like noise. It's hard to explain what the 3D plot actually looks like; basically it's useless.

I'm wondering if there are certain requirements to be met; such as lighting, reflection from surfaces, etc that if any of them gets messed up the toolbox will just spit out some trash data. I would appreciate any tips on what should be taken care of!

Answer 1

[Dima Lisin]

Hi Mahdi,

Here are some tips for you:

1. Camera Calibration: Please make sure you have good stereo calibration. Make sure that your calibration images are in focus, and that you have enough images to cover most of the field of view. Use the showReprojectionErrors to check the reprojection errors. Ideally, they should be less than .5 of a pixel. If you see that any calibration image pairs produce reprojection errors much higher than the others, try excluding them and re-calibrating. Also, look at the result of rectfyStereoImages to make sure that the rectification makes sense. Correspoinding points should be on the same horizontal lines.
2. Pre-processing Rectified Images: Look carefully at your disparity map. Can you see the shapes of the objects of interest? Is there too much noise? If the disparity map looks bad, try pre-processing the rectified images. Applying histogram equalization using the histeq function can help. It may also help to low-pass filter the rectified images.
3. Tuning Disparity Parameters: Try tuning the parameters of the disparity function. Specifically 'Disparity Range' and 'Block Size'. To figure out what the disparity range should be create an anaglyph of the rectified images using the stereoAnaglyph function, and display it using imtool. In imtool measure the distances in pixels between a few pairs of corresponding points in the two images. That should give you an idea of what the disparity range should be. Then tune 'Block Size' to get the object silhouettes to appear in the disparity map without too much noise. If your disparity values are very high (e. g. greater than 256 pixels), then you can try moving the cameras closer together, or moving the cameras further away from the objects of interest. If your disparity values are close to 0, then you should either move the cameras further apart from each other, or move them closer to the objects of interest.
4. Post-processing Disparity Map: Once you get a decent disparity map, you can try to clean up the noise some more by applying the median filter to it using the medfilt2 function.
5. Restricting the 3D Volume: Once you get the 3-D points from reconstructScene, you can get rid of much of the noise by "cropping" the 3-D volume. Simply eliminate the points for which the Z coordinate is either too small (e.g. less than 0) or too large, by setting their coordinates to NaN. You can similarly limit the range of X and Y.

[Safwana Razak]

hi Dima,

Just wonder, is the mentioned workflow/tips applied for 3D reconstruction light? where the input is 1 standard camera together with light projector. Something like this video: 3D structured scanner

I'm confuse how to calibate the camera with the projector light. Are we able to use Camera Calibrator app (for single camera) with the checkerbox with the projector light?

and how to perform the reconstruction for that input, if stereo camera we just use disparity, disparityBM or disparitySGM?

and for the visualization, are we going to use reconstructScene() and use any pointcloud visualization function to display the 3D object?

thanks