RNAscope staining of breast cancer tissue allows pathologists to deduce genetic characteristics of the cancer by inspection at the microscopic level, which can lead to better diagnosis and treatment. Chromogenic RNAscope staining is easy to fit into existing pathology workflows, but manually analysing the resulting tissue samples is time consuming. In addition, there is a lack of verified supporting methods for quantification and analysis. This thesis covers the development of methods to annotate and process image data from whole slide images of tissue microarrays, and then to accurately segment and quantify the RNAscope dots (each representing a single RNA transcript) from this tissue.

We first developed a method to automatically find and label each core in a tissue microarray grid from a whole slide image, which achieved a precision of 99.63%. Following this, we created a dataset of 480x480 pixel breast cancer tissue patches, and had the RNAscope dots on these patches annotated by two experts using a custom QuPath script. This dataset provided both training data for development of further methods and sufficient data to produce a baseline expert inter-rater agreement score. The expert inter-rater agreement, as assessed by F1-score, was 0.596.

Next, we investigated the usefulness of grey level texture features for automatically segmenting and classifying the positions of RNAscope dots from breast cancer tissue. Feature analysis showed that a small set of grey level features, including Grey Level Dependence Matrix and Neighbouring Grey Tone Difference Matrix features, were well suited for the task. This feature-based method performed similarly to expert annotators at identifying the positions of RNAscope dots, with an F1-score of 0.571.

Finally, we developed and optimized a novel deep learning method focused on accurate segmentation of RNAscope dots from breast cancer tissue. The deep learning network is convolutional, using ConvNeXt as its backbone. The upscaling portions of the network use custom, heavily regularized blocks to prevent overfitting and early convergence on suboptimal solutions. The resulting network is modest in size for a segmentation network, and able to function well with little training data. This deep learning network was also able to outperform manual expert annotation at finding the positions of RNAscope dots, having a final F1-score of 0.745. This final score was obtained after in-depth analysis and optimization of multiple model parameters, and exploration of artificial data generation to improve training performance.

The methods developed and analysed in this thesis provide an accurate, automated pipeline for quantification of RNAscope that could be integrated into the pathology workflow.