Within the frames of this study, the synthesis of a permalloy to be used as a filler for magnetoactive and magnetorheological elastomers (MAEs and MREs) was carried out. By means of the mechanochemical method, an alloy with the composition 75 wt.% of Fe and 25 wt.% of Ni was obtained. The powder of the product was utilized in the synthesis of MAEs. Study of the magnetorheological (MR) properties of the elastomer showed that in a ~400 mT magnetic field the shear modulus of the MAE increased by a factor of ~200, exhibiting an absolute value of ~8 MPa. Furthermore, we obtained experimentally a relative high loss factor for the studied composite; this relates to the size and morphology of the synthesized powder. The composite with such properties is a very perspective material for magnetocontrollable damping devices. Under the action of an external magnetic field, chain-like structures are formed inside the elastomeric matrix, which is the main determining factor for obtaining a high MR effect. The effect of chain-like structures formation is most pronounced in the region of small strains, since structures are partially destroyed at large strains. A proposed theoretical model based on chain formation sufficiently well describes the experimentally observed MR effect. The peculiarity of the model is that chains of aggregates of particles, instead of individual particles, are considered.