In this article, we present a novel Lagrangian particle tracking method derived from the perspective of the tracking-by-detection paradigm that has been adopted by many vision tracking tasks. Under this paradigm, the particle tracking problem consists of first learning a function (the tracker) that maps the target particle’s image projection backwardly to its possible position inferred from its precedent tracking information. The target particle’s actual position is then detected by applying the learned function to particle images captured by cameras. We also propose to solve the function learning problem using kernel methods. The proposed method is therefore named Kernelized Lagrangian particle tracking (KLPT). The current state-of-art LPT approach Shake-The-Box (STB), despite equipping a highly efficient image matching and shaking-based optimization procedure, tends to be trapped by local minimum when dealing with challenging cases featuring sparse temporal data, data extracted from complex flows and noisy data. KLPT can overcome these optimization difficulties since it features a highly robust function learning procedure combined with an efficient linear optimization technique. We assessed our proposed KLPT against various STB implementations both on synthetic and real experimental datasets. For the synthetic dataset depicting a turbulent cylinder wake-flow at Re=3900, we focused on studying the effects of particle density, time separation, and image noise. KLPT outperformed STB in all cases by tracking more particles and producing more accurate particle fields. This performance gain, compared to STB, is more prominent for the dataset with larger seeding density, time separation, and more noise. For real experimental data on an impinging jet flow in a water tank, KLPT can capture longer tracks and provides more detailed flow reconstruction at highly turbulent regions than STB. Overall, comparison to STB shows significant improvements in accuracy (lower reconstruction positional error) and robustness (more and longer tracks). We finally show the results of KLPT on the 1st LPT challenge datasets. Our algorithm has achieved state-of-the-art performance with particle images up to 0.08 ppp (particles per pixel).