This thesis addresses the detection problem in array signal processing in two aspects: (a) detection problems in white noise environments; (b) detection problems in unknown coloured (spatially correlated) noise environments. New criteria for determining the number of signals in both these kinds of noise environments are developed. The performance of the new methods is analyzed theoretically and is confirmed by computer simulations using Monte Carlo method. The status of the existing methods for detection in array processing are reviewed. For the white noise environment, some unfavourable characteristics of existing methods are discussed, for example, the subjective threshold setting required by the traditional threshold methods, and the rigid performance of the information theoretic criteria. A new method, namely Eigen-Threshold (ET) method is proposed and analyzed theoretically and checked by computer simulations. The new method demonstrates superiority over the existing methods by: (a) not requiring a subjective threshold setting as required by the traditional threshold methods; (b) possessing a flexible performance which can be easily controlled by a single parameter, in contrast to the rigid performance given by the information theoretic criteria. By properly choosing the control parameter, the new method gives better performance than both AIC and MDL. Because of these advantages, the new ET method is more applicable in practice than other existing methods. Besides enjoying the same merit of not requiring a subjective threshold setting, the ET method gives a quantitative controllable performance which is useful in practice, because although the asymptotic consistency argument used in information theoretic criteria and some other methods has important theoretical significants but: (a) in any practical application the sample size can only be a limited number; (b) when the sample size N is given, and a quantitative performance is desired, the asymptotic consistency argument may not make too much sense since such arguments could not give even an approximate error level except predicting whether the error rate will go to zero when N goes to infinity. For the more difficult detection problem in the case of spatially correlated noise, there has not been any satisfactory method developed so far. By assuming a banded structure for the noise covariance matrix, which is true for many engineering applications, and applying a bi-array structure combined with canonical correlation analysis, a new elegant method is developed in this thesis. The new method, called Canonical Correlation Test (CCT) method, gives a reliable, simple, theoretically sound solution to the detection problem in unknown coloured noise environments. Massive simulations have shown that the new method is extremely robust to changes in the noise spectrum. Again, the new method is characterized by a quantitatively controllable performance. To compare the new methods with the existing methods, the widely accepted AIC and MDL criteria are used for comparison purpose through out this thesis.

Doctor of Philosophy (PhD)