This correspondence presents a video tracking framework using control-based observer design. It unifies several kernel-based approaches into a consistent theoretical framework by modeling tracking as a recursive inverse problem. The framework relies on observability theory to handle the "singularity" problem and provides explicit criteria for kernel design and dynamics evaluation.