The static and homogeneous nature of the existing state-of-the-art networked systems provides asymmetric advantages to attackers that make them easy for reconnaissance and launching the attacks. The advanced cyberattacks (e.g., APT, DDoS, malware) cause tremendous socioeconomic impact and losses; therefore, there is an immediate need to not only respond to the already happened attacks but disrupt the attackers for preventing the attacks. The concept of moving target defense (MTD) is to dynamically change attack surface to increase uncertainty and confuse the attackers by invalidating the attacker’s intelligence or information collected during the reconnaissance or procedures to launch the attacks. Software-defined networking (SDN) has emerged as a promising technology that provides flexibility and programmability to networked systems, which facilitates the implementation of the MTD operations for cybersecurity. MTD research based on SDN environments is in an infant stage since both MTD and SDN research areas have emerged relatively recently.

Identifying hosts and vulnerable services (e.g., IP addresses and TCP/UDP ports) of the target systems is a precursory step for the vast majority of the cyberattacks. Network address space randomization, random host mutation, and ports hopping are commonly used network address shuffling MTD approaches. However, there are problems in these existing MTD approaches. First, these MTD mechanisms randomize IP addresses or port numbers of a host mapping them to a virtual IP or port in a one-to-one manner, which requires more IP addresses to satisfy the mutation rate and

unpredictability constraints. Secondly, they lack to use both IP address and ports shuffling so that the services are vulnerable since they are long-time exposure to potential attackers. Thirdly, the existing security metrics are limited to particular attack scenarios or models, very application-specific, and lack to capture and measure the effectiveness of the various types of MTD techniques (e.g., shuffling of IP, ports, or network topology). This thesis aims to address the aforementioned problems in three primary research goals: (1) to develop a moving target defense mechanism that continuously and dynamically changes the virtual IP addresses of a server host in the SDN; (2) to design and develop a moving target defense mechanism that protects the SDN by shuffling both hosts’ virtual IP addresses and services’ port numbers in the SDN; and (3) to develop a set of dynamic security metrics for measuring the effectiveness of the SDN-based MTD techniques.

To achieve the goal (1), a new MTD approach Flexible Random Virtual IP Multiplexing, namely FRVM, is proposed, which aims to thwart network reconnaissance and scanning attacks in software-defined networks. FRVM enables a host machine to have multiple, random, time-varying, virtual IP addresses, which are multiplexed to a real IP address of the host. The FRVM frequently changes all the virtual IP addresses of the host over time; the multiplexing and de-multiplexing operations of it remaps the virtual IP to a real IP, and a real IP to the virtual IP address(s) of the host respectively. Therefore, at the end of every multiplexing event, the FRVM aims to make the attackers lose any knowledge gained through the reconnaissance and to disturb their scanning strategy. The performance evaluation of the FRVM is comparatively analyzed with a baseline model (i.e., a typical static network without FRVM mechanism) through simulations and experiments using the security (e.g., attacker success probability) and performance metrics (e.g., delay, throughput). The results show that the proposed FRVM effectively increases the attacker’s work effort with acceptable operational overhead.

To achieve the goal (2), a new proactive, dynamic SDN-based MTD mechanism Random Host and Service Multiplexing, namely RHSM, is developed. RHSM uses the multiplexing or de-multiplexing of both IP addresses (e.g., hosts) and port numbers (e.g., services) aiming to obfuscate both network and transport layers’ real identities of the hosts and the services for defending against the network reconnaissance and scanning attacks. RHSM allows each host to use random, multiple virtual IP addresses to be dynamically and periodically shuffled. Also, it uses short-lived, multiple virtual port numbers for active services running on the host. The RHSM dynamically and frequently changes all the virtual IPs and the virtual ports of the host and services, respectively. The simulations and experimental results demonstrate the RHSM effectively outperform a baseline counterpart in terms of the attack success probability and defense cost.

To achieve the goal (3), new dynamic security metrics are developed. The proposed dynamic security metrics that timely, dynamically, and adaptively assess the effectiveness of the SDN-based MTD techniques. The security metrics are developed to measure the dynamics of a network and a host state’s information (e.g., IP address, port, software stacks, vulnerabilities, or network topology) introduced by various types of MTD techniques shuffling them. The key aspect of our proposed metrics is to capture variability that keeps track of changing patterns of the network and the host states upon every MTD triggering event. The proposed metrics are following three categories: (i) Network and host address-based metrics that measure variability of the network and the host addresses based on a degree of uncertainty and unpredictability on the assigned IP address to the hosts in a network; (ii) Attack path-based metrics which are used to measuring variability of attack paths using graphical models estimated based on the network state transitions from one topology to another topology upon triggering a network topology and/or IP shuffling MTD; and (iii) attack stage-based success metrics measure the chances of discovering a vulnerable target host’s information, exploiting the target host’s vulnerability, and compromising the target host. Via extensive simulation

study, the key parameters that can significantly affect the performance of the MTD are investigated using the proposed security metrics.

In summary, the main contributions of this thesis are: (1) the development of the flexible random virtual IP multiplexing (FRVM) in the SDNs; (2) the evaluation of the security performance and overhead of the MTD in the SDNs; (3) the development of the random host and service multiplexing (RHSM) for MTD in the SDNs; and (4) the development of the dynamic security metrics for measuring effectiveness of the MTDs in the SDNs.