Intervention fidelity, the extent to which an intervention is delivered, as intended, is rarely examined in “real-world” settings. This project aimed to establish whether Media Smart, a media literacy program intended to help prevent eating disorders, is delivered as intended by its developers, by developing a psychometrically valid and reliable rating scale to assess this. Media Smart-NZ was delivered in five New Zealand schools to 11–12-year-olds, by a range of facilitators with no background in eating disorders. These lessons were audio recorded and adherence to the key principles of Media Smart, and competence in delivery of the program were rated using a randomly selected sample of these recordings. The psychometric properties of the rating scale, Media Smart-NZ Assessment of Facilitator Competence (MS-FidelityNZ), were examined and the overall fidelity of the Media Smart NZ intervention delivery was evaluated. The MS-FidelityNZ was found to have good internal consistency (Cronbach’s alpha 0.89) and good inter rater reliability scores. ICC was between 0.76 and 0.84 (0.559 – 0.93), depending on the inclusion or omission of the one dichotomous choice item. An exploratory factor analysis revealed two underlying constructs; 1. effective delivery of the key principles of Media Smart and 2. the facilitator not delivering proscribed content. Overall fidelity to the program was moderate to high, with a mean total score of 40 (SD 9.16), with total scores ranging from 23 - 55. Comparisons between facilitators revealed a spread of mean scores from 24 (SD = 0.78) to 52 (SD = 4.32), indicating a between facilitator difference in adherence to and competence in delivery of the key Media Smart principles. This is the first rating scale of its kind, developed to evaluate fidelity to the Media Smart program and its results indicate that it may be a useful tool to help ensure that the real-world delivery of Media Smart is monitored, with the resulting information used to adequately train a range of facilitators.