Context. Existing fluid simulation methods have several disadvantages and can be improved with the help of new approaches to the solution of problems of computational fluid dynamics, which confirms the relevance of the work. Objective. The goal of the work is to improve existing methods of mathematical modeling of fluid based on smoothed particle hydrodynamics and R-functions method. Method. A new approach of joint use of smoothed particle hydrodynamics, marching cubes and R-functions method is proposed. Smoothed particle hydrodynamics helps to simulate fluid movement in real time. The method considers fluid as a discrete number of sample points (particles), which have mass, velocity, position and physical field quantities (pressure, temperature, mass-density, etc.). The R-functions method allows to solve the inverse problem of analytic geometry: finding an analytical equation of a 2D (3D) object based on its geometrical representation. Using the obtained equation, one can simply detect a particle collision with the object boundary and plot the object surface with the help of marching cubes algorithm. The suggested method allows to achieve good simulation quality and to perform all needed calculations and rendering in real time. Results. Computational experiments for the problem of fluid simulation were carried out. Various numbers of particles were used. Different kinds of objects were put into the considered region in order to investigate the fluid behavior. Conclusions. The results of visual simulations allow us to say that the obtained approach works as expected. Therefore, this method can be applied to several problems of fluid simulation where the collision detection with arbitrary objects is considered. Further research may be devoted to the optimization of neighbor-search algorithm, to performing all calculations in graphics processing unit or to taking into account other physical quantities.