Recent work by scientists at the University of Waterloo and Darwin AI, a Toronto-based AI company, provides new metrics to measure the trustworthiness of deep learning systems in an intuitive and interpretable way. Trust is often a subjective issue, but their research, presented in two papers, provides clear guidelines on what to look for when evaluating the scope of situations in which AI models can and can\u2019t be trusted.<\/p>\n
[Read: How to build a search engine for criminal data<\/span><\/a>]<\/em><\/p>\n For many years, machine learning researchers measured the trustworthiness of their models through metrics such as accuracy, precision, and F1 score. These metrics compare the number of correct and incorrect predictions made by a machine learning model in various ways. They can answer important questions such as if a model is making random guesses or if it has actually learned something. But counting the number of correct predictions doesn\u2019t necessarily tell you whether a machine learning model is doing its job correctly.<\/p>\n More recently, the field has shown a growing interest in <\/span>explainability<\/a>, a set of techniques that try to interpret decisions made by <\/span>deep neural networks<\/a>. Some techniques highlight the pixels that have contributed to a deep learning model\u2019s output. For instance, if your convolutional neural network has classified an image as \u201csheep,\u201d explainability techniques can help you figure out whether the neural network has learned to detect sheep or if it is classifying patches of grass as sheep.<\/p>\n Explainability techniques can help you make sense of how a deep learning model works, but not when and where it can and can\u2019t be trusted.<\/p>\n In their <\/span>first paper<\/a>, titled, \u201cHow Much Can We Really Trust You? Towards Simple, Interpretable Trust Quantification Metrics for Deep Neural Networks,\u201d the AI researchers at Darwin AI and the University of Waterloo introduce four new metrics for \u201cassessing the overall trustworthiness of deep neural networks based on their behavior when answering a set of questions.\u201d<\/p>\n While there are other papers and research work on measuring trust, these four metrics have been designed to be practical for everyday use. On the one hand, the developers and users of AI systems should be able to continuously compute and use these metrics to constantly monitor areas in which their deep learning models can\u2019t be trusted. On the other hand, the metrics should be simple and interpretable.<\/p>\n In the <\/span>second paper<\/a>, titled, \u201cWhere Does Trust Break Down? A Quantitative Trust Analysis of Deep Neural Networks via Trust Matrix and Conditional Trust Densities,\u201d the researchers introduce the \u201ctrust matrix,\u201d a visual representation of the trust metrics across different tasks.<\/p>\n Consider two types of people, one who is overly confident in their wrong decisions and another who is too hesitant about the right decision. Both would be untrustworthy partners. We all like to work with people who have a balanced behavior: they should be confident about their right answers and also know when a task is beyond their abilities.<\/p>\n In this regard, machine learning systems are not very different from humans. If a neural network classifies a stop sign as a speed limit sign with a 99 percent confidence score, then you probably shouldn\u2019t install it in your self-driving car. Likewise, if another neural network is only 30 percent confident it is standing on a road, then it wouldn\u2019t help much in driving your car.<\/p>\n \u201cQuestion-answer trust,\u201d the first metric introduced by the researchers, measures an AI model\u2019s confidence in its right and wrong answer. Like classical metrics, it takes into account the number of right and wrong predictions a machine learning model makes, but also factors in their confidence scores to penalize overconfidence and overcautiousness.<\/p>\n Say your machine learning model must classify nine photos and determine which ones contain cats. The question-answer trust metric will reward every right classification by a factor of its confidence score. So obviously, higher confidence scores will receive a higher reward. But the metric will also reward wrong answers by the inverse of the confidence score (i.e., 100% \u2013 confidence score). So a low confidence score in a wrong classification can earn as much reward as high confidence in the right classification.<\/p>\n What\u2019s interesting about this metric is that, unlike precision and accuracy scores, it\u2019s not about how many right predictions your machine learning model makes\u2014after all, nobody is perfect. It\u2019s rather about how trustworthy the model\u2019s predictions are.<\/p>\n Question-answer trust enables us to measure the trust level of single outputs made by our deep learning models. In their paper, the researchers expand on this notion and provide three more metrics that enable us to evaluate the overall trust level of a machine learning model.<\/p>\n The first, \u201ctrust density,\u201d measures the trust level of a model on a specific output class. Say you have a neural network trained to detect 20 different types of pictures, but you want to measure its overall trust level in the class \u201ccat.\u201d Trust density visualizes the distribution question-answer trust of the machine learning model for \u201ccat\u201d across multiple examples. A strong model should show higher density toward the right (question-answer trust = 1.0) and lower density toward the left (question-answer trust = 0.0).<\/p>\n The second metric, \u201ctrust spectrum,\u201d further zooms out and measures the model\u2019s trustworthiness across different classes when tested on a finite set of inputs. When visualized, the trust spectrum provides a nice overview of where you can and can\u2019t trust a machine learning model. For instance, the following trust spectrum shows that our neural network, in this case ResNet-50, can be trusted in detecting teapots and school buses, but not in screens and monitors.<\/p>\n Finally, the \u201cNetTrustScore\u201d summarizes the information of the trust spectrum into a single metric. \u201cFrom an interpretation perspective, the proposed NetTrustScore is fundamentally a quantitative score that indicates how well placed the deep neural network\u2019s confidence is expected to be under all possible answer scenarios that can occur,\u201d the researchers write.<\/p>\n In their complementary paper, the AI researchers introduce the trust matrix, a visual aid that gives a quick glimpse of the overall trust level of a machine learning model. Basically, the trust matrix is a grid that maps the outputs of a machine learning model to their actual values and the trust level. The vertical axis represents the \u201coracle,\u201d the known values of the inputs provided to the machine learning model. The horizontal axis is the prediction made by the model. The squares represent a test, with the X-axis representing the output by the model and the Y-axis representing its actual value. The color of each square shows its trust level, with bright colors representing high and dark colors representing low trust.<\/p>\n A perfect model should have bright-colored squares across the diagonal going from the top-left to the bottom-right, where predictions and ground truth cross paths. A trustworthy model can have squares that are off the diagonal, but those squares should be colored brightly as well. A bad model will quickly show itself with dark-colored squares.<\/p>\n For instance, the red circle represents a \u201cswitch\u201d that was predicted as a \u201cstreet sign\u201d by the machine learning model with a low trust score. This means that the model was very confident it was seeing a street sign while in reality, it was looking at a switch. On the other hand, the pink circle represents a high trust level on a \u201cwater bottle\u201d that was classified as a \u201claptop.\u201d This means the machine learning model had provided a low confidence score, signaling that it was doubtful of its own classification.<\/p>\n The hierarchical structure of the trust metrics proposed in the papers makes them very useful. For instance, when choosing a machine learning model for a task, you can shortlist your candidates by reviewing their NetTrustScores and trust matrices. You can further investigate the candidates by comparing their trust spectrums on multiple classes and further compare their performance on single classes on the trust density score.<\/p>\n The trust metrics will help you quickly find the best model for your task or find important areas where you can make improvements to your model.<\/p>\n Like many areas of machine learning, this is a work in progress. In their current form, the machine learning trust metrics only apply to a limited set of supervised learning problems, namely classification tasks. In the future, the researchers will be expanding on the work to create metrics for other kinds of tasks such as object detection, speech recognition, and time series. They will also be exploring trust in unsupervised machine learning algorithms.<\/p>\n \u201cThe proposed metrics are by no means perfect, but the hope is to push the conversation towards better quantitative metrics for evaluating the overall trustworthiness of deep neural networks to help guide practitioners and regulators in producing, deploying, and certifying deep learning solutions that can be trusted to operate in real-world, mission-critical scenarios,\u201d the researchers write.<\/p>\n This article was originally published by Ben Dickson on <\/span><\/i>TechTalks<\/span><\/i><\/a>, a publication that examines trends in technology, how they affect the way we live and do business, and the problems they solve. But we also discuss the evil side of technology, the darker implications of new tech and what we need to look out for. You can read the original article here<\/a>.<\/span><\/i><\/p>\n Published December 2, 2020 \u2014 11:00 UTC <\/p>\nHow far do you trust machine learning?<\/h2>\n
![\"confusion](\"https:\/\/i1.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/11\/confusion-matrix.jpg?resize=696%2C171&ssl=1\")
![\"RISE](\"https:\/\/i1.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2018\/10\/RISE-explainable-AI-example-saliency-map.png?resize=696%2C524&ssl=1\")
Overconfident of overcautious?<\/h2>\n
![\"cats](\"https:\/\/i1.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/11\/cats-and-dogs.jpg?resize=600%2C600&ssl=1\")
Setting up a hierarchy of trust scores<\/h2>\n
![\"trust](\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/11\/image.png?resize=640%2C480&ssl=1\")
![\"Trust](\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/11\/image-1.png?resize=696%2C232&ssl=1\")
The machine learning trust matrix<\/h2>\n
![\"Trust](\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/11\/image-2.png?resize=696%2C557&ssl=1\")
<\/figure>\n<\/div>\nPutting trust metrics to use<\/h2>\n
![\"comparing](\"https:\/\/i0.wp.com\/bdtechtalks.com\/wp-content\/uploads\/2020\/11\/comparing-metrics.jpg?resize=696%2C523&ssl=1\")