Imputation of missing data using masked denoising autoencoder with L2-norm regularization in software effort estimation - Marco, Robert; Syed Ahmad, Sharifah Sakinah (Intelligent Network and Systems Society, 2024)
A frequent problem in building initial software effort estimation (SEE) models is the existence of many missing values in historical software engineering datasets. Due to human intervention, this is caused by frequent damage to software project data. Loss of information and bias in data analysis due to missing data are serious problems. This study proposes a method to estimate missing data using a masked-denoising autoencoder (MaskedDAE) with L2-norm regularization, which can handle various types of data, missing patterns, proportions, and distributions. In this study, Cocomo81 and ISBSG-IFPUG datasets from open-source repositories were used. This experiment involved five missing data techniques, eight missing data rates (from 10% to 80%), and two missingness mechanisms (MCAR: missing completely at random and MNAR: missing not at random). The results show that the proposed Mask-DAE method has the best imputation performance in terms of imputation errors by outperforming DAE, k-nearest neighbor imputation (kNNI), random forest (RF) imputation, multiple imputations by chained equation (MICE), mean imputation and mode imputation. We find that the prediction error rate increases with the rate of missing data. Furthermore, prediction errors generated by MCAR mechanisms are lower than those generated by MNAR. Nevertheless, our method can reduce the model variance, which results in lower generalization error.