From a combination of UltraVISTA and 3D-HST data we measure the fractions of quiescent, unobscured star-forming, and dusty star-forming galaxies as a function of stellar mass since z = 3. Motivated by this study, we investigate the stellar and dust properties of a mass complete sample of massive and dusty galaxies at 1 ≤ z ≤ 4 by modeling their UV-IR spectral energy distributions (SEDs) obtained from UltraVISTA DR3 photometry and Herschel PACS-SPIRE data using MAGPHYS. We evaluate the ability of the rest-frame UVJ color-color diagram to determine the star formation and dust obscuration properties for our sample. We construct median SEDs as functions of redshift and star-formation level. Simultaneous modeling of the panchromatic SED allows us to quantify the contribution to the IR emission from dust heated by star formation rather than evolved stellar populations, which we find to be crucial to characterize these galaxies correctly. We finally compare the z > 1 sample to local galaxies.