The high-spatial-resolution nature of optical non-invasive measurement and the reconstruction of three-dimensional sound fields are important for evaluating acoustic transducers. The computed tomography (CT) method has been used to reconstruct sound fields from optical measurement results because of its contactless feature. However, since the CT method does not assume any physical property of sound, it may not be suitable for the reconstruction of sound fields. In contrast, a physical-model-based method is able to reconstruct sound fields more accurately by considering the physics of sound. In this study, we propose a scan-free and physical-model-based reconstruction of a three-dimensional sound field. A sound field is multi-directionally measured using parallel phase-shifting interferometry and a high-speed polarization camera by rotating an acoustic transducer. A physical-model-based method is applied to reconstruct the sound field from the recorded optical interferograms. We also discussed the effectiveness of the proposed method to evaluate acoustic transducers.