Active Video Watching (AVW-Space) is an online platform for video-based learning which supports engagement via note-taking and personalized nudges. In this paper, we focus on the quality of the comments students write. We propose two schemes for assessing the quality of comments. Then, we evaluate these schemes by computing the inter-coder agreement. We also evaluate various machine learning classifiers to automate the assessment of comments. The selected cost-sensitive classifier shows that the quality of comments can be assessed with high weighted-F1 scores. This study contributes to the automation of comment quality assessment and the development of personalized educational support for engagement in video-based learning through commenting.