We present the first time-domain adjoint variable method (AVM) algorithm for materials with dispersive constitutive parameters. We develop our algorithm based on transmission-line modeling techniques for electromagnetic problems. The developed theory is based on utilizing the Z-domain representation of the dispersive materials, which can model arbitrary dispersive behavior. We develop a formulation similar to the original AVM theory for nondispersive materials. The theory has been successfully applied to problems with dispersive materials modeled by the Drude, Debye, and Lorentz models.