The electron beam-plasma system is ubiquitous in the space plasma environment. Here, using a Darwin particle-in-cell method, the excitation of electrostatic and whistler instabilities by a gyrating electron beam is studied in support of recent laboratory experiments. It is assumed that the total plasma frequency (ωpe) is larger than the electron cyclotron frequency (Ωe). The fast-growing electrostatic beam-mode waves saturate in a few plasma oscillations by slowing down and relaxing the electron beam parallel to the background magnetic field. Upon their saturation, the finite amplitude electrostatic beam-mode waves can resonate with the tail of the background thermal electrons and accelerate them to the beam parallel velocity. The slower-growing whistler waves are excited in primarily two resonance modes: (a) through Landau resonance due to the inverted slope of the beam electrons in the parallel velocity and (b) through cyclotron resonance by scattering electrons to both lower pitch angles and smaller energies. It is demonstrated that, for a field-aligned beam, the whistler instability can be suppressed by the electrostatic instability due to a faster energy transfer rate between the beam electrons and electrostatic waves. Such a competition of growth between whistler and electrostatic waves depends on the ratio of ωpe/Ωe. In terms of wave propagation, beam-generated electrostatic waves are confined to the beam region, whereas beam-generated whistler waves transport energy away from the beam.