The determination of linguistic structure from surface patterns in text and speech requires the integration of cues from multiple constraint domains including phonetic features, syllable and morpheme structure, syntax, and semantics together with pragmatics. Utilization of these constraints shows that the factors contributing to the integration metric vary along the utterance, and that principled surface variation can be accounted for in terms of these structures, thus reducing the apparent noise. Given the large number of factors that influences the pattern classification decision, it is important to defer commitment to structural hypotheses as long as possible, so that neither “bottom up” nor “top down” search strategies are appropriate models for the recognition of natural language patterns. Instead, observance of cooccurrence relations among the parameters of a model can be exploited in efficient training procedures that extract the maximum amount of information from the experimental corpus. These techniques naturally lead to formulations of constraint domain structures that are mathematically explicit, minimizing the use of heuristics except where dictated by complexity considerations. Experience from contemporary research in speech synthesis and recognition is used to illustrate these principles, characterize current capability, and indicate directions for future research.