Significant progress has been made by researchers in image classification mainly due to the accessibility of large-scale public visual datasets and powerful Convolutional Neural Network(CNN) models. Pre-trained CNN models can be utilized for learning comprehensive features from smaller training datasets, which support the transfer of knowledge from one source domain to different target domains. Currently, there are numerous frameworks to handle image classifications using transfer learning including preparing the preliminary features from the early layers of pre-trained CNN models, utilizing the mid-/high-level features, and fine-tuning the pre-trained CNN models to work for different targeting domains. In this work, we proposed to build a genetic algorithm-based deep learning model selection framework to address various detection challenges. This framework automates the process of identifying the most relevant and useful features generated by pre-trained models for different tasks. Each model differs in numerous ways depending on the number of layers.