Visual object tracking from unmanned aerial vehicles (UAVs) demands both high accuracy and computational efficiency for real-time deployment on resource-constrained platforms. While state space models (SSMs) offer linear computational complexity, existing methods face critical deployment challenges. They rely on the HiPPO framework with complex discretization procedures and employ hardware-aware algorithms optimized for high-performance GPUs, which introduce deployment overhead and are difficult to transfer to edge platforms. Additionally, their fixed polynomial bases may cause information loss for tracking features with complex geometric structures. We propose LF-SSM, a lightweight HiPPO (High-order Polynomial Projection Operators)-free state space model that reformulates state evolution on Riemannian manifolds. The core contribution is the Geodesic State Module (GSM), which performs state updates through tangent space projection and exponential mapping on the unit sphere. This design eliminates complex discretization and specialized hardware kernels while providing adaptive local coordinate systems. Extensive experiments on UAV benchmarks demonstrate that LF-SSM achieves state-of-the-art performance while running at 69 frames per second (FPS) with only 18.5 M parameters, demonstrating superior efficiency for real-time edge deployment.