In this work wireless communication systems with multiple co-channel signals present at the receiver are considered. One of the major challenges in the development of such systems is the computational complexity required for the detection of the transmitted signals. This thesis addresses this problem and develops reduced complexity algorithms for the detection of multiple co-channel signals in receivers with multiple antennas. The signals are transmitted from either a single user employing multiple transmit antennas, from multiple users or in the most general case by a mixture of the two. The receiver is assumed to be overloaded in that the number of transmitted signals exceeds the number of receive antennas. Joint Maximum Likelihood (JML) is the optimum detection algorithm which has exponential complexity in the number of signals. As a result, detection of the signals of interest at the receiver is challenging and infeasible in most practical systems.

The thesis presents a framework for the detection of multiple co-channel signals in overloaded receivers. It proposes receiver structures and two list-based signal detection algorithms that allow for complexity reduction compared to the optimum detector while being able to maintain near optimum performance. Complexity savings are achieved by first employing a linear preprocessor at the receiver to reduce the effect of Co-Channel Interference (CCI) and second, by using a detection algorithm that searches only over a subspace of the transmitted symbols. Both algorithms use iterative processing to extract ordered lists of the most likely transmit symbols. Soft information can be obtained from the detector output list and can then be used by error control decoders.

The first algorithm named Parallel Detection with Interference Estimation (PD-IE) considers the Additive White Gaussian Noise (AWGN) channel. It relies on a spatially reduced search over subsets of the transmitted symbols in combination with CCI estimation. Computational complexity under overload is lower than that of JML. Performance results show that PD-IE achieves near optimum performance in receivers with Uniform Circular Array (UCA) and Uniform Linear Array (ULA) antenna geometries.

The second algorithm is referred to as List Group Search (LGS) detection. It is applied to overloaded receivers that operate in frequency-flat multipath fading channels. The List Group Search (LGS) detection algorithm forms multiple groups of the transmitted symbols over which an exhaustive search is performed. Simulation results show that LGS detection provides good complexity-performance tradeoffs under overload.

A union bound for group-wise and list-based group-wise symbol detectors is also derived. It provides an approximation to the error performance of such detectors without the need for simulation. Moreover, the bound can be used to determine some detection parameters and tradeoffs. Results show that the bound is tight in the high Signal to Interference and Noise Ratio (SINR) region.