A deep learning based approach for image captioning: Exploiting priority factor to enhance accuracy and relevance

The image captioning task is regarded as an active problem in computer vision research. The focal aim of this task is to generate accurate descriptions for the content of an input image. The state-of-the-art automatic image captioningsystems typically rely on the Encoder-Decoder architecture and only a few among which exploit the features resulting fromthe object detection task in order to maximize the accuracy of captions generation. In this work we introduce a new attentionguided Encoder-Decoder based captioning approach that utilizes two types of features: a) deep visual features extracted from EfficientNetV2 pre-trained model on ImageNet dataset, and b) object features extracted from YOLOv7 pre-trained model on MSCOCO dataset. Additionally, we compute a new object features-driven feature called Priority Factor as a utility to rank objects based on their prominence in input images. The proposed approach is evaluated using a famous dataset in computer vision (MSCOCO). The empirical performance of our approach is measured using eight metrics, and the results elaborate the effectiveness of adding our schema (Priority Factor) to object features, leading to minor improvements in evaluation metrics BLEU-1, BLEU-2, BLEU-3, and BLEU-4. This approach outperforms four state of the art approaches in various evaluation metrics such as BLEU-1, BLEU-2, BLEU-3, and SPICE.