It is known that there are no local scalar Lie fields in more than two dimensions. Bilocal fields, however, which naturally arise in conformal operator product expansions, do generate infinite Lie algebras. It is demonstrated that these Lie algebras of local observables admit (highly reducible) unitary positive energy representations in a Fock space. The multiplicity of their irreducible components is governed by a compact gauge group. The mutually commuting observable algebra and gauge group form a dual pair in the sense of Howe. In a theory of local scalar fields of conformal dimension two in four space-time dimensions the associated dual pairs are constructed and classified.The paper reviews joint work of B. Bakalov, N. M. Nikolov, K.-H. Rehren and the author.