Users’ trust in black-box machine learning algorithms
DOI:
https://doi.org/10.1108/REGE-06-2022-0100Keywords:
Artificial intelligence, Black-box systems, Machine learning, Trust, ExplainabilityAbstract
PurposeThis paper aims to investigate whether professional data analysts’ trust of black-box systems is increased by explainability artifacts.
Design/methodology/approachThe study was developed in two phases. First a black-box prediction model was estimated using artificial neural networks, and local explainability artifacts were estimated using local interpretable model-agnostic explanations (LIME) algorithms. In the second phase, the model and explainability outcomes were presented to a sample of data analysts from the financial market and their trust of the models was measured. Finally, interviews were conducted in order to understand their perceptions regarding black-box models.
FindingsThe data suggest that users’ trust of black-box systems is high and explainability artifacts do not influence this behavior. The interviews reveal that the nature and complexity of the problem a black-box model addresses influences the users’ perceptions, trust being reduced in situations that represent a threat (e.g. autonomous cars). Concerns about the models’ ethics were also mentioned by the interviewees.
Research limitations/implicationsThe study considered a small sample of professional analysts from the financial market, which traditionally employs data analysis techniques for credit and risk analysis. Research with personnel in other sectors might reveal different perceptions.
Originality/valueOther studies regarding trust in black-box models and explainability artifacts have focused on ordinary users, with little or no knowledge of data analysis. The present research focuses on expert users, which provides a different perspective and shows that, for them, trust is related to the quality of data and the nature of the problem being solved, as well as the practical consequences. Explanation of the algorithm mechanics itself is not significantly relevant.
Downloads
References
Adadi, A., & Berrada, M. (2018). Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access, 6, 52138–52160.
Adams, B. D., Bruyn, L. E., & Houde, S. (2003). Trust in automated systems (report). Ministry of National Defence.
Barocas, S., Friedler, S., Hardt, M., Kroll, J., Venka-Tasubramanian, & Wallach, H. (2018). The FAT-ML workshop series on fairness, accountability, and transparency in machine learning. available from: http://www.fatml.org/
Brynjolfsson, E., & McAfee, A. (2017). Artificial intelligence, for real. Harvard Business Review. Available from: https://hbr.org/cover-story/2017/07/the-business-of-artificial-intelligence.
Cahour, B., & Forzy, J.-F. (2009). Does projection into use improve trust and exploration? An example with a cruise control system. Safety Science, 47(9), 1260–1270. doi: 10.1016/j.ssci.2009.03.015.
Chen, Y. W., Yang, J. B., Pan, C. C., Xu, D. L., & Zhou, Z. J. (2015). Identification of uncertain nonlinear systems: Constructing belief rule-based models. Knowledge-Based System, 73, 124–133.
Confalonieri, R., Weyde, T., Besold, T. R., & Moscoso del Prado Martín, F. (2021). Using ontologies to enhance human understandability of global post-hoc explanations of black-box models. Artificial Intelligence, 296, 1–20. doi: 10.1016/j.artint.2021.103471.
Dikmen, M., & Burns, C. (2022). The effects of domain knowledge on trust in explainable AI and task performance: A case of peer-to-peer lending. International Journal of Human-Computer Studies, 162, 1–11. doi: 10.1016/j.ijhcs.2022.102792.
Farquad, M. A. H., Ravi, V., & Raju, S. B. (2012). Analytical CRM in banking and finance using SVM: A modified active learning-based rule extraction approach. International Journal of Electronic Customer Relationship Management, 6(1), 48–60. doi: 10.1504/ijecrm.2012.046470.
Gléonnec, M. (2004). Confiance et usage des technologies d’information et de communication. Consommations et Sociétés, 4, 1–18.
Hoffman, R. R., Mueller, S. T., Klein, G., & Litman, J. (2018). Metrics for explainable AI: Challenges and prospects, ARXIV, 1-50. doi: 10.48550/arXiv.1812.04608.
Kenny, E. M., Ford, C., Quinn, M., & Keane, M. T. (2021). Explaining black-box classifiers using post-hoc explanations-by-example: The effect of explanations and error-rates in XAI user studies. Artificial Intelligence, 294, 1–25. doi: 10.1016/j.artint.2021.103459.
Lewis, P. R., & Marsh, S. (2022). What is it like to trust a rock? A functionalist perspective on trust and trustworthiness in artificial intelligence. Cognitive Systems Research, 72, 33–49. doi: 10.1016/j.cogsys.2021.11.001.
Lu, H., Li, Y., Chen, M., Hyoungseop, K., & Serikawa, S. (2018). Brain intelligence: Go beyond artificial intelligence. Mobile Networks and Applications, 23(2), 368–375. doi: 10.1007/s11036-017-0932-8.
Moreira, C., Chou, Y.-L., Velmurugan, M., Ouyang, C., Sindhgatta, R., & Bruza, P. (2021). LINDA-BN: An interpretable probabilistic approach for demystifying black-box predictive models. Decision Support Systems, 150, 1–16. doi: 10.1016/j.dss.2021.113561.
Narwaria, M. (2022). Does explainable machine learning uncover the black box in vision applications?. Image and Vision Computing, 118, 1–4. doi: 10.1016/j.imavis.2021.104353.
O’Neil, C. (2017). Weapons of math destruction: How big data increases inequality and threatens democracy. New York: Crown Publishing Group.
Pereira, S., Meier, R., McKinley, R., Wiest, R., Alves, V., Silva, C. A., & Reyes, M. (2017). Enhancing interpretability of automatically extracted machine learning features: Application to a RBM-random forest system on brain lesion segmentation. Medical Image Analysis, 44, 228–244. doi:10.1016/j.media.2017.12.009.
Preece, A. (2018). Asking ‘why’ in AI: Explainability of intelligent systems—perspectives and challenges. Intelligent Systems in Accounting, Finance and Management, 25, 63–72.
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). ‘Why should i trust you?’ Explaining the predictions of any classifier. Proceedings of the International Conference on Knowledge Discovery and Data Mining, 1135–1144. doi: 10.1145/2939672.2939778.
Setzu, M., Guidotti, R., Monreale, A., Turini, F., Pedreschi, D., & Giannotti, F. (2021). GLocalX - from local to global explanations of black box AI models. Artificial Intelligence, 294, 1–24. doi: 10.1016/j.artint.2021.103457.
Shin, D. (2021). The effects of explainability and causability on perception, trust, and acceptance: Implications for explainable AI. International Journal of Human-Computer Studies, 146, 1–10. doi: 10.1016/j.ijhcs.2020.102551.
van Lent, M., Fisher, W., & Mancuso, M. (2004). An explainable artificial intelligence system for small-unit tactical behavior. Proceedings of the 16th Conference on Innovative Applications of Artificial Intelligence (pp. 900–907). San Jose, CA: AAAI Press.
Wang, F., & Rudin, C. (2015). Falling rule lists. Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, 38, 1013–1022.
