Video-mediated communication (VMC) is currently the prevalent mode of telecommunication for applications such as remote collaboration, teleconferencing, and distance learning. It is generally assumed that transmitting real-time talking-head videos of participants in addition to their audio is beneficial and desirable, enabling remote conferencing to feel almost the same as face-to-face collaboration. However, compared to being face-to-face, VMC still feels distant, artificial, cumbersome, and detached. One limitation of standard video-collaboration that contributes to this feeling is that the 3D context between people and their shared workspace given in face-to-face collaboration is lost. It is therefore not possible for participants to tell from the video what others are looking at, what they are working on, or who they are talking to. Video Collaborative Virtual Environments (video-CVEs) are novel VMC interfaces which address these problems by re-introducing a virtual 3D context into which distant users are mentally "transported" to be together and interact with the environment and with each other, represented by their spatially controllable video-avatars. To date, research efforts following this approach have primarily focused on the demonstration of working prototypes. However, maturation of these systems requires a deeper understanding of human factors that emerge during mediated collaborative processes. This thesis contributes to a deeper understanding of human factors. It investigates the hypothesis that video-CVEs can effectively support face-to-face aspects of collaboration which are absent in standard video-collaboration. This hypothesis is tested in four related comparative user studies involving teams of participants collaborating in video-CVEs, through standard video-conferencing systems, and being face-to-face. The experiments apply and extend methods from the research fields of human-computer interaction, computer-supported cooperative work, and presence. Empirical findings indicate benefits of video-CVEs for user experience dimensions such as social presence and copresence, but also highlight challenges for awareness and usability that need to be overcome to unlock the full potential of this type of interface.