This work presents a non-convex variational approach to joint image reconstruction and labeling. Our regularization strategy, based on the KL-divergence, takes into account the smooth geometry on the space of discrete probability distributions. The proposed objective function is efficiently minimized via DC programming which amounts to solving a sequence of convex programs, with guaranteed convergence to a critical point. Each convex program is solved by a generalized primal dual algorithm. This entails the evaluation of a proximal mapping, evaluated efficiently by a fixed point iteration. We illustrate our approach on few key scenarios in discrete tomography and image deblurring.