Irregularity of a graph is an invariant measuring how much the graph differs from a regular graph. Albertson index is one measure of irregularity, defined as the sum of | deg(u) - deg(v) | over all edges uv of the graph. Motivated by a recent result on the irregularity of Fibonacci cubes, we consider irregularity polynomials and determine them for Fibonacci and Lucas cubes. These are graph families that have been studied as alternatives for the classical hypercube topology for interconnection networks. The irregularity polynomials specialize to the Albertson index and also provide additional information about the higher moments of | deg(u) - deg(v) | in these families of graphs.