The purpose of this paper is to determine the group of all automorphisms of the field generated by automorphic functions with respect to infinitely many mutually commensurable discrete subgroups of the group of all automorphisms of a bounded symmetric domain. In the case where either the dimension of the domain is one or the quotient spaces are compact, and also in some other cases, the automorphism groups have been determined by Ihara [4], Pjateckii-Shapiro and Shafarevic [6], and Shimura [7]. We will determine the group of all automorphisms under the assumption that the quotient spaces of the domain by discrete groups are isomorphic to algebraic varieties (not necessarily complete or non-singular) and the maps of the algebraic varieties corresponding to the natural projection maps of the quotient spaces are algebraic. By virtue of the result of Baily and Borel [2], this condition is satisfied if the discrete subgroups are arithmetic subgroups of a semi-simple algebraic group. Since useful applications arise mainly in the arithmetic cases, the reader may assume, without losing much substance, that the discrete groups are arithmetic subgroups of a semi-simple algebraic group. I would like to express my deep gratitude to Professor G. Shimura and Professor A. Borel for their reading the entire manuscript and for giving many useful suggestions, and to Professor J-I. Igusa for valuable conversations which helped to simplify parts of the proof.