Passive 3D Time-of-Flight (ToF) imaging faces a significant challenge in accurate depth recovery, where the source position is unavailable. In this letter, we present a probabilistic approach based on the Kalman filter which keeps track of source location, thus, avoiding the need for an initial guess to jointly determine the 3D source position and correct depth information. The proposed approach is able to reach a source location error of 0.8 cm by exploiting pseudo-measurements of the plane fitting constraint acquired by a passive ToF camera which exploits a bistatic algorithm with a gradient descent method. Computer experiments are carried out to demonstrate the robustness of the proposed method in realistic scenarios where no initial guess is available. The results show the feasibility of blind source localization with mm accuracy. This contributes to widening the applicability of passive 3D imaging in a number of application fields, such as autonomous driving, and robot navigation.