Speaker
Description
In supervised Machine Learning (ML) or Deep Learning (DL) projects, a model is trained, validated and tested by selecting the optimal preprocessing parameters, hyperparameters, and model architecture. The model’s performance is then optimized based on the preferred performance metric, such as accuracy or F1-score. In most cases, the number and distribution of the input data is kept fixed. However, when new data is added, the distribution changes over time and in turn the model performance degenerates. To reach the original performance, the model needs to be updated, by re-running the training pipeline.
With Continuous Machine Learning (CML) the model performance can be monitored with dedicated behavioral tests as the characteristics of the data might change over time. Additionally, these behavioral tests provide insights on the performance of corner cases or especially difficult examples. This can be helpful, if the so far best model needs to be updated based on evaluation metrics regarding the whole data set. Although, the overall performance of the successor model has improved, it might have a degenerated performance on certain corner cases you otherwise do not directly become aware of.
In our work, we train a Wasserstein GANs (wGAN) with FIRST radio galaxy images to generate additional radio galaxy images because the number of available labeled images is limited. The idea is to improve a classifier by adding additional generated images to the training data set besides the real images. In this case, the training data set is changing with every new version of the generator. Thus, with every new version of the generator, the question is how much the generated images improved the classifier and what are the reasons for the improvement. Therefore, we implemented automated behavioral tests that always run when the training data set changes due to new versions of the generator. For one behavioral test, we selected radio galaxy sources from the test data set and used their labels, but we acquired the images from a different type of radio telescope named LOFAR. Thus, the wGAN and the classifier are not provided with these images in their training and validation data sets. To gain more insights on how the generated images might improve the classifier, we used a Vision Transformer (ViT) as a classifier. The ViT provides attention maps that indicate on which part of the image the classifier bases its decision on. This gives a better understanding on what the model is focusing on and how that changes with additional generated images.
The results show that we can improve the classifier with additional generated images, especially in one class, and the attention maps indicate that the classifier is focusing on the expected structures within the image. Overall, the combination of automated behavioral tests and the ViT as classifier continuously provides insights about the current best model in the iterative development of the wGAN and classifier.
