FCNs are a model for real-time neural network for class-wise image segmentation. As the name implies, every weight layer in the network is convolutional. The final layer has the same height/width as the input image, making FCNs a useful tool for doing dense pixel-wise predictions without a significant amount of postprocessing. Being fully convolutional also provides great flexibility in the resolutions this model can handle.
This specific model detects 20 different classes. The models have been pre-trained on the COCO train2017 dataset on this class subset.
The input is expected to be an image with the shape (N, 3, height, width) where N is the number of images in the batch, and height and width are consistent across all images.
Preprocessing
The images must be loaded in RGB with a range of [0, 1] per channel, then normalized per-image using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225].
This model can take images of different sizes as input. However, it is recommended that the images are resized such that the minimum size of either edge is 520.
The following code shows an example of how to preprocess a demo image:
The model has two outputs, ("out", "aux"). "out" is the main classifier and has shape (N, 21, height, width). Each output pixel is one-hot encoded, i.e. np.argmax(out[image, :, x, y]) is that pixel's predicted class. Class 0 is the background class.
"aux" is an auxilliary classifier with the same shape performing the same functionality. The difference between the two is that "out" sources features from last layer of the ResNet backbone, while "aux" sources features from the second-to-last layer.
Postprocessing steps
The following code shows how to overlay the segmentation on the original image:
Pretrained weights from the Torchvision Model Zoo were used instead of training these models from scratch. A conversion notebook is provided.
Validation accuracy
Mean IoU (intersection over union) and global pixelwise accuracy are computed on the COCO val2017 dataset.
Torchvision reports these values as follows:
The more conservative of the two estimates is used in the model files table.
Compared with the fp32 FCN ResNet 50, FCN ResNet 50-int8's mean IoU drop ratio is 0.46% global pixelwise accuracy drop ratio is 0.10% and performance improvement is 1.28x.
Note
The performance depends on the test hardware. Performance data here is collected with Intel® Xeon® Platinum 8280 Processor, 1s 4c per inst ance, CentOS Linux 8.3, data batch size is 1.
Quantization
FCN ResNet 50-int8 and FCN ResNet-50-qdq are obtained by quantizing fp32 FCN ResNet 50 model. We use Intel® Neural Compressor with onnxruntime backend to perform quantization. View the instructions to understand how to use Intel® Neural Compressor for quantization.
Make sure to specify the appropriate dataset path in the configuration file.
bash
1bash run_tuning.sh --input_model=path/to/model \# model path as *.onnx2--config=fcn_rn50.yaml \3--data_path=path/to/coco/val2017 \4--label_path=path/to/coco/annotations/instances_val2017.json \5--output_model=path/to/save
References
Jonathan Long, Evan Shelhamer, Trevor Darrell; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3431-3440
This model is converted from the Torchvision Model Zoo, originally implemented by Francisco Moss here.
