Fifth generation (5G) cellular networks must support dense user environments where close proximity users and their associated highly correlated channel vectors are prevalent. Therefore, this thesis considers a multi-user multiple-input multiple-output (MU-MIMO) communication system, where both highly correlated users and semiorthogonal users are present. Specifically, this thesis provides a performance analysis of several established and novel linear signal processing schemes for users with highly correlated channels. The main focus of this thesis is divided into two scenarios.

First, we have proposed two novel schemes namely decorrelating zero forcing (DZF) and hybrid zero forcing (HZF), which are robust in user environments where highly correlated users are present. DZF simply decorrelates the channels of two highly correlated users exploiting the advantage of mutual orthogonality of eigenvectors of highly correlated users, i.e., DZF employs the first and second eigenvectors while designing the precoders for two highly correlated users. DZF is as simple as conventional zero forcing (CZF), but achieves higher rates in highly correlated channel environments. Analysis and numerical results are presented for MU-MIMO systems demonstrating the impact of the DZF scheme, which improves user rates and provides fairness while scheduling the highly correlated users. Then, a more robust HZF scheme is designed by integrating the CZF and DZF schemes. In CZF, semiorthogonal users are scheduled to achieve higher sum-rate, meaning that fairness among correlated users is compromised. HZF allows us to harness the advantages of both CZF and DZF, hence providing robustness against the joint scheduling of semi-orthogonal and highly correlated users with very little additional complexity.

Second, we studied the existing downlink non-orthogonal multiple access zero- forcing beamforming (NOMA-ZFBF) system that exploits the high correlation between users. Furthermore, we propose a new power allocation scheme that ensures the weak user’s quality of service (W-QoS) in a pair of strong and weak highly correlated users. Assuming an arbitrary correlation among the clustered users we derive the signal to noise ratio (SNR) and signal to interference plus noise ratio (SINR) expressions of clustered strong and weak users. These expressions contribute towards two important insights; a) the required correlation of clustered users and b) the required power differential between the highly correlated users in a cluster. Further, assuming perfectly correlated users we derive the exact SNR/SINR distribution of strong users, weak users and singletons. On the basis of these distributions we compute a closed-form approximation for the expected sum-rate of the NOMA system that gives an upper-bound on system performance. This is useful to understand how the correlation factor affects the clustering rate and how close partially correlated channels can get to the perfectly correlated bound. Numerical results confirm the reliability of the proposed W-QoS NOMA based techniques and lead to the important insights required to understand the performance of NOMA in 5G cellular communication.

Finally, the variance of the interference power between two user channels in a downlink Rician fading channel with spatial correlation at the Base Station (BS) is analyzed. We show that high variance is beneficial to NOMA so that NOMA performance is directly linked to the variance of the interference power. This leads to three key properties linking channel statistics to NOMA potential, namely; a) similar channel statistics increase the variance, b) equal correlation matrices increase the variance and c) similar line of sight (LoS) directions increase the variance. Comparison results presented between correlated and uncorrelated Rayleigh and Rician channels suggest that highest variance can be obtained in correlated Rayleigh channels with a narrow angular spread which yields higher cluster percentage, inter-cluster correlation, and cluster-rate.

In summary, we can conclude that the DZF scheme can achieve a higher rate than CZF in highly correlated user environments and HZF is robust to the user environment where both similar channel users and orthogonal channel users co-exist. Furthermore, when ZFBF is employed in NOMA systems, then a power differential among the highly correlated clustered users is not required. Finally, correlated Rayleigh channels having similar channel statistics are most suitable for NOMA-ZFBF due to the higher variance it provides for the interference power.