Polarization vectors of light traveling in a coiled optical fiber rotate around its propagating axis even in the absence of birefringence. This rotation was usually explained due to the Pancharatnam-Berry phase of spin-1 photons. Here, we use a purely geometric method to understand this rotation. We show that similar geometric rotations also exist for twisted light carrying orbital angular momentum (OAM). The corresponding geometric phase can be applied in photonic OAM-state-based quantum computation and quantum sensing.