Let $G$ be a graph and $A$ be its adjacency matrix. A graph $G$ is invertible if its adjacency matrix $A$ is invertible and the inverse of $G$ is a weighted graph with adjacency matrix $A^{-1}$. A signed graph $(G,σ)$ is a weighted graph with a special weight function $σ: E(G)\to \{-1,1\}$. A graph is sign-invertible (or sign-invertible) if its inverse is a signed graph. A sign-invertible graph is always unimodular. The inverses of graphs have interesting combinatorial interests. In this paper, we study inverses of graphs and provide a combinatorial description for sign-invertible graphs, which provides a tool to characterize sign-invertible graphs. As applications, we complete characterize sign-invertible bipartite graphs with a unique perfect matching, and sign-invertible graphs with cycle rank at most two. As corollaries of these characterizations, some early results on trees (Buckley, Doty and Harary in 1982) and unicyclic graphs with a unique perfect matching (Kalita and Sarma in 2022) follow directly.