In this paper, we propose a theoretical model to study the elastic response of a granular material idealized as a random aggregate of identical, elastic, frictional spheres that has been isotropically compressed. When we consider that contacting particles move according to the average deformation, the effective shear modulus is over-predicted with respect to numerical simulation, while the effective bulk modulus is almost captured. We improve upon this simple approach by relaxing the aggregate. The kinematics of a pair of contacting particles is then given by the average deformation and fluctuations in both translations and rotations. We determine analytical expressions for these fluctuations by means of force and moment equilibrium applied to each particle of the pair. In order to derive the incremental stress associated with an incremental deformation, we introduce conditional averages of the fluctuations that are functions of the statistical geometry of the packing. This brings the theoretical predictions of the effective moduli close to those measured in numerical simulations. The variability of the average number of particle contacts per particle is seen to play an important role in the statistical description of the aggregate.