We propose a new bivariate distribution following a GLM form i.e., natural exponential family given the constantly correlated covariance matrix. The proposed distribution can represent an independent bivariate gamma distribution as a special case. In order to derive the distribution we utilize an integrating factor method to satisfy the integrability condition of the quasi-score function. The derived distribution becomes a mixture of discrete and absolute continuous distributions. The proposal of our new bivariate distribution will make it possible to develop some bivariate generalized linear models. Further the discrete correlated bivariate distribution will also arise from an independent bivariate Poisson mass function by compounding our proposed distribution (Iwasaki and Tsubaki, 2002).