The segmentation model was first trained on ImageNet (Deng et al., 2009), and then the model was fine-tuned on a specific set of image data relevant to the domain: Illinois Natural History Survey Fish Collection (INHS Fish).
The Feature Pyramid Network (FPN) architecture was used for fine-tuning, since it is a CNN-based architecture designed to handle multi-scale feature maps (Lin et al., 2017: IEEE, arXiv).
The FPN uses SE-ResNeXt as the base network (Hu et al., 2018: IEEE, arXiv).
Model Description
PyTorch implementation of Fish trait segmentation model.
This segmentation model is based on pretrained model using the segementation models torch.
Then the model is fine tuned on fish images in order to identify (segment) the different traits.
To increase the size and diversity of the training dataset (originally 295 images), we employed data augmentation techniques such as flipping, shifting, rotating, scaling, and adding noise to the original image data to increase the dataset 10-fold.
We developed 12 target classes, or trait masks, for our segmentation problem, each representing different morphological traits of a fish specimen.
The segmentation classes are: dorsal fin, adipose fin, caudal fin, anal fin, pelvic fin, pectoral fin, head minus the eye, eye, caudal fin-ray, alt fin-ray, alt fin-spine, and trunk.
Although minnows do not have adipose fins, the segmentation model was trained on a variety of fish image data, some of which had adipose fins.
We retained this class because the segmentation model may erroneously assign an adipose fin to a minnow (Fig. S1), and a domain scientist examining these outputs may want to analyze the accuracy of the model.
The segmentation has 13 classes: background (black), dorsal fin (red), adipose fin (green), caudal fin (blue), anal fin (yellow), pelvic fin (light blue), pectoral fin (pink), head minus the eye (white), eye (bright green), trunk (teal), caudal fin-ray (light red), alt fin-ray (light pink), and alt fin-spine (peach).
We prepared the model by using the Segmentation Model PyTorch library (Iakubovskii, 2019) to load an FPN segmentation model that was pretrained on the Imagenet dataset.
We used SE-ResNeXt as the base network/encoder to extract features (embedding) from the input image data and replaced the last decoder layer with 12 target classes.
During the fine-tuning procedure, the encoder of the pre-trained model was frozen as these layers already contain useful features that we can leverage.
We only tuned the decoder weights of our segmentation model during this fine-tuning procedure.
We then trained the prepared model for 120 epochs, updating the weights using dice loss as a measure of similarity between the predicted and ground-truth segmentation.
The Adam optimizer (Kingma & Ba, 2014) with a small learning rate (1e-4) was used to update the model weights.
Evaluation
We evaluated the performance of the fine-tuned segmentation model on the test set using the Intersection over Union (IoU) score with a 0.5 threshold.
(The IoU score ranges from 0 to 1, with 1 indicating a perfect overlap between the predicted segmentation and the ground-truth segmentation and 0 indicating no overlap.)
Our segmentation model achieved a 0.90 mIoU score on the test dataset.
Testing Data, Factors & Metrics
We had 99 testing images and 98 validation images.
More Information
Research supported by NSF Office of Advanced Cyberinfrastructure (OAC) Awards #2022042, #1940233, #1940322, and #1940247, with additional support from NSF Award #2118240.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
