School dropout is a widely recognized serious issue in developing countries. On the other hand, machine learning techniques have gained much attention on addressing this problem. This paper, presents a thorough analysis of four supervised learning classifiers that represent linear, ensemble, instance and neural networks on Uwezo Annual Learning Assessment datasets for Tanzania as a case study. The goal of the study is to provide data-driven algorithm recommendations to current researchers on the topic. Using three metrics: geometric mean, F-measure and adjusted geometric mean, we assessed and quantified the effect of different sampling techniques on the imbalanced dataset for model selection. We further indicate the significance of hyper parameter tuning in improving predictive performance. The results indicate that two classifiers: logistic regression and multilayer perceptron achieve the highest performance when over-sampling technique was employed. Furthermore, hyper parameter tuning improves each algorithm's performance compared to its baseline settings and stacking these classifiers improves the overall predictive performance.