Abstract: Most existing intelligent fault diagnosis studies focus on improving accuracy, implying that decisions are made only by models. From the safety aspect, this over-reliance on models can lead to users having no way of knowing even if the model gives untrustworthy diagnostic results; from the ethical aspect, the current artificial intelligence (AI) technology lacks moral guidance, and the relevant laws are not yet perfect, so it is difficult to pursue responsibility in case of misdiagnosis. A reliable diagnosis model should not only provide as accurate results as possible, but should also point out the possibility of its decision failure to warn the user. Therefore, it is necessary to assess the confidence of the results to mitigate the risk of model failure and to achieve trustworthy fault diagnosis. However, modern deep learning models are often poorly calibrated, i.e., there is a mismatch between the softmax output, which is often considered to characterize the confidence of the result, and the true probability of the result being correct, leading to a significant bias in using it directly as a confidence level. To this end, we propose a calibration technique called adaptive confidence penalty that fine-tunes the strength of the confidence penalty applied to each training sample, which in turn affects the softmax probability of the validation/testing samples inferred by the model. The method compensates for the limitation of the original confidence penalty method that uses a fixed penalty strength without considering the confidence characteristics of each sample, further improving the calibration quality and obtaining well-calibrated diagnosis models. The experimental results illustrate the motivation for designing the proposed method and demonstrate its superiority.

Key words: model calibration, confidence estimation, trustworthy artificial intelligence, rotating machinery fault diagnosis