Modern networked systems are complex in such a way that assessing the security of them is a difficult task. Security models are widely used to analyse the security of these systems, which are capable of evaluating the complex relationship between network components. Security models can be generated by identifying vulnerabilities, threats (e.g., cyber attacks), network configurations, and reachability of network components. These network components are then combined into a single model to evaluate how an attacker may penetrate through the networked system. Further, countermeasures can be enforced to minimise cyber attacks based on security analysis. However, modern networked systems are becoming large sized and dynamic (e.g., Cloud Computing systems). As a result, existing security models suffer from scalability problem, where it becomes infeasible to use them for modern networked systems that contain hundreds and thousands of hosts and vulnerabilities. Moreover, the dynamic nature of modern networked systems requires a responsive update in the security model to monitor how these changes may affect the security, but there is a lack of capabilities to efficiently manage these changes with existing security models. In addition, existing security models do not provide functionalities to capture and analyse the security of unknown attacks, where the combined effects of both known and unknown attacks can create unforeseen attack scenarios that may not be detected or mitigated. Therefore, the three goals of this thesis are to (i) develop security modelling and analysis methods that can scale to a large number of network components and adapts to changes in the networked system; (ii) develop efficient security assessment methods to formulate countermeasures; and (iii) develop models and metrics to incorporate and assess the security of unknown attacks.

A lifecycle of security models is introduced in this thesis to concisely describe performance and functionalities of modern security models. The five phases in the lifecycle of security models are: (1) Preprocessing, (2) Generation, (3) Representation, (4) Evaluation, and (5) Modification.

To achieve goal (i), a hierarchical security model is developed to reduce the computational costs of assessing the security while maintaining all security information, where each layer captures different security information. Then, a comparative analysis is presented to show the scalability and adaptability of security models. The complexity analysis showed that the hierarchical security model has better or equivalent complexities in all phases of the lifecycle in comparison to existing security models, while the performance analysis showed that in fact it is much more scalable in practical network scenarios.

To achieve goal (ii), security assessment methods based on importance measures are developed. Network centrality measures are used to identify important hosts in the networked systems, and security metrics are used to identify important vulnerabilities in the host. Also, new network centrality measures are developed to improvise the lack of accuracy of existing network centrality measures when the attack scenarios consist of attackers located inside the networked system. Important hosts and vulnerabilities are identified using efficient algorithms with a polynomial time complexity, and the accuracy of these algorithms are shown as nearly equivalent to the naive method through experiments, which has an exponential complexity.

To achieve goal (iii), unknown attacks are incorporated into the hierarchical security model and the combined effects of both known and unknown attacks are analysed. Algorithms taking into account all possible attack scenarios associated with unknown attacks are used to identify significant hosts and vulnerabilities. Approximation algorithms based on dynamic programming and greedy algorithms are also developed to improve the performance. Mitigation strategies to minimise the effects of unknown attacks are formulated on the basis of significant hosts and vulnerabilities identified in the analysis. Results show that mitigation strategies formulated on the basis of significant hosts and vulnerabilities can significantly reduce the system risk in comparison to randomly applying mitigations.

In summary, the contributions of this thesis are: (1) the development and evaluation of the hierarchical security model to enhance the scalability and adaptability of security modelling and analysis; (2) a comparative analysis of security models taking into account scalability and adaptability; (3) the development of security assessment methods based on importance measures to identify important hosts and vulnerabilities in the networked system and evaluating their efficiencies in terms of accuracies and performances; and (4) the development of security analysis taking into account unknown attacks, which consists of evaluating the combined effects of both known and unknown attacks.