diff --git a/chapter_explainable_AI/explainable_ai.md b/chapter_explainable_AI/explainable_ai.md index 1d4b421..9ab99f9 100644 --- a/chapter_explainable_AI/explainable_ai.md +++ b/chapter_explainable_AI/explainable_ai.md @@ -41,7 +41,6 @@ - 可解释AI是确保人类可以轻松理解和信任人工智能代理做出的决策的一组方法。 -可见，关注点在于对模型的理解、黑盒模型白盒化以及模型的可信任。 我们根据自身的实践经验和理解，将可解释AI定义为：一套面向机器学习（主要是深度神经网络）的技术合集，包括可视化、数据挖掘、逻辑推理、知识图谱等，目的是通过此技术合集，使深度神经网络呈现一定的可理解性，以满足相关使用者对模型及应用服务产生的信息诉求（如因果或背景信息），从而为使用者对人工智能服务建立认知层面的信任。 @@ -61,18 +60,18 @@ 全局方法旨在提供对模型逻辑的理解以及所有预测的完整推理，基于对其特征、学习到的组件和结构的整体视图等等。有几个方向可以探索全局可解释性。为了便于理解，我们将它们分为以下三个子类： (i) -模型提取——从原始黑盒模型中提取出一个可解释的模型，比如通过模型蒸馏的方式将原有黑盒模型蒸馏到可解释的决策树，从而使用决策树中的规则解释该原始模型； +模型提取——从原始黑盒模型中提取出一个可解释的模型，比如通过模型蒸馏的方式将原有黑盒模型蒸馏到可解释的决策树 :cite:`frosst2017distilling` :cite:`zhang2019interpreting`，从而使用决策树中的规则解释该原始模型； (ii) 基于特征的方法——估计特征的重要性或相关性，如图 :numref:`xai_global_feature_importance`所示, -该类型解释可提供如“信用逾期记录是模型依赖的最重要特征”的解释，从而协助判定模型是否存在偏见 -(iii) 透明模型设计——修改或重新设计黑盒模型以提高其可解释性。 +该类型解释可提供如“信用逾期记录是模型依赖的最重要特征”的解释，从而协助判定模型是否存在偏见. 一种典型的全局特征解释方法是SHAP（其仅能针对树模型输出全局解释）:cite:`lundberg2017unified`。 +(iii) 透明模型设计——修改或重新设计黑盒模型以提高其可解释性。这类方法目前也逐渐成为探索热点，近期的相关工作包括ProtoPNet :cite:`chen2019looks`, Interpretable CNN :cite:`zhang2018interpretable`, ProtoTree :cite:`nauta2021neural`等。 ![全局特征重要性解释](../img/ch11/xai_global_feature_importance.png) :width:`800px` :label:`xai_global_feature_importance` -全局解释可以提供黑盒模型的整体认知。但由于黑盒模型的高复杂性，在实践中往往很难通过模型提取/设计得到与原模型行为相近的简单透明模型，也往往很难对整个数据集抽象出统一的特征重要性。此外，在为单个观察生成解释时，全局解释也缺乏局部保真度，因为全局重要的特征可能无法准确解释单个样例的决定。因此，局部方法成为了近些年领域内重要的研究方向。局部方法尝试为单个实例或一组实例检验模型行为的合理性。当仅关注局部行为时，复杂模型也可以变得简单，因此即使是简单的函数也有可以为局部区域提供可信度高的解释。基于获得解释的过程，局部方法可以分为两类：局部近似和基于梯度传播的方法。 +全局解释可以提供黑盒模型的整体认知。但由于黑盒模型的高复杂性，在实践中往往很难通过模型提取/设计得到与原模型行为相近的简单透明模型，也往往很难对整个数据集抽象出统一的特征重要性。此外，在为单个观察生成解释时，全局解释也缺乏局部保真度，因为全局重要的特征可能无法准确解释单个样例的决定。因此，局部方法成为了近些年领域内重要的研究方向。局部方法尝试为单个实例或一组实例检验模型行为的合理性。当仅关注局部行为时，复杂模型也可以变得简单，因此即使是简单的函数也有可以为局部区域提供可信度高的解释。基于获得解释的过程，局部方法可以分为两类：局部近似和基于传播的方法。 局部近似是通过在样本近邻区域模拟黑盒模型的行为生成可理解的子模型。相比于全局方法中的模型提取，局部近似仅需关注样本临近区域，因此更容易获得精确描述局部行为的子模型。如图 :numref:`xai_lime`所示，通过在关注数据点$x$附近生成$m$个数据点$(x_i^\prime, f(x_i^\prime)), for\ i=1,2, ...m$（这里$f$为黑盒模型决策函数）,用线性拟合这些数据点，可以得到一个线性模型$g=\sum_i^k w_ix^i$，这里$k$表示数据的特征维度。那么线性模型中的权重$w_i$即可用于表示数据$x$中第$i$个特征对于模型$f$的重要性。 @@ -80,8 +79,9 @@ :width:`800px` :label:`xai_lime` -基于梯度传播的方法通常是用梯度传播直接定位相关特征，这些方法封装了基于反向传播的方法和基于前向传播的方法。基于反向传播的方法将输出的贡献归因于输入特征。如图 :numref:`xai_gradient_based`所示,通过梯度回传，计算模型输出对输入的梯度$\frac{d(f(x)}{dx}$ 作为模型解释。 -而基于前向传播的方法通过扰动特征后的输出差异来量化输出与特征的相关性。 +基于传播的方法通常是传播某些信息直接定位相关特征，这些方法包含了基于反向传播的方法和基于前向传播的方法。基于反向传播的方法通过梯度回传将输出的贡献归因于输入特征。如图 :numref:`xai_gradient_based`所示,通过梯度回传，计算模型输出对输入的梯度$\frac{d(f(x)}{dx}$ 作为模型解释。常见的基于梯度传播的方法有基本Gradient方法，GuidedBackprop :cite:`zeiler2014visualizing`, GradCAM :cite:`selvaraju2017grad`等. +而基于前向传播的方法通过扰动特征后, 进行前向推理的输出差异来量化输出与特征的相关性。其中，常见的几种方法有RISE :cite:`petsiuk2018rise`，ScoreCAM :cite:`wang2020score`等。 + ![局部近似方法示例](../img/ch11/xai_gradient_based.PNG) :width:`800px` @@ -110,7 +110,7 @@ TCAV就可以通过计算类$k$的具有正$S_{C,k,l}$’s的样本的比率来 $$\textbf{TCAV}_{Q_{C,k,l}}=\frac{\vert \{\mathbf{x}\in X_{k}:S_{C,k,l}(\mathbf{x})>0\}\vert}{\vert X_{k}\vert} \label{eq:TCAV}$$ -结合$$t$$-分布假设方法，如果$$\textbf{TCAV}_{Q_{C,k,l}}$$大于0.5，则表明概念$$C$$对类$$k$$有重大影响。 +结合$t$-分布假设方法，如果$\textbf{TCAV}_{Q_{C,k,l}}$大于0.5，则表明概念$C$对类$k$有重大影响。 ![TCAV流程(图片来源于 :cite:`2020tkde_li`)](../img/ch11/xai_tcav.png) :width:`800px` @@ -179,13 +179,13 @@ saliency = grad_cam(boat_image, targets=3) :width:`400px` :label:`correct_correct` -上图预测标签是“bird”，解释结果给出依据的关键特征在鸟身上，说明这个分类判断依据是合理的, 可以初步判定模型为可信的。 +上图预测标签是“bicycle”，解释结果给出依据的关键特征 在车轮上，说明这个分类判断依据是合理的, 可以初步判定模型为可信的。 ![预测结果正确，依据的关键特征不合理的例子](../img/ch11/correct_wrong.png) :width:`400px` :label:`correct_wrong` -原图中，有人，在预测标签中有1个标签是“person”，这个结果是对的；但是解释的时候，高亮区域在马头的上，那么这个关键特征依据很可能是错误的, 这个模型的可靠性还需进一步验证。 +上图在预测标签中有1个标签是“person”，这个结果是对的；但是解释的时候，高亮区域在马头的上，那么这个关键特征依据很可能是错误的, 这个模型的可靠性还需进一步验证。 ![预测结果错误，依据的关键特征不合理的例子](../img/ch11/wrong_wrong.png) :width:`400px` diff --git a/references/explainable.bib b/references/explainable.bib index ff76626..511c0bf 100644 --- a/references/explainable.bib +++ b/references/explainable.bib @@ -1,59 +1,154 @@ @ARTICLE{2020tkde_li, author={Li, Xiao-Hui and Cao, Caleb Chen and Shi, Yuhan and Bai, Wei and Gao, Han and Qiu, Luyu and Wang, Cong and Gao, Yuanyuan and Zhang, Shenjia and Xue, Xun and Chen, Lei}, - journal={IEEE Transactions on Knowledge and Data Engineering}, - title={A Survey of Data-driven and Knowledge-aware eXplainable AI}, + journal={IEEE Transactions on Knowledge and Data Engineering}, + title={A Survey of Data-driven and Knowledge-aware eXplainable AI}, year={2020}, volume={}, number={}, pages={1-1}, - doi={10.1109/TKDE.2020.2983930} -} - + doi={10.1109/TKDE.2020.2983930}} + @article{erhan2009visualizing, - title={Visualizing higher-layer features of a deep network}, - author={Erhan, Dumitru and Bengio, Yoshua and Courville, Aaron and Vincent, Pascal}, - journal={University of Montreal}, - volume={1341}, - number={3}, - pages={1}, - year={2009} +author = {Erhan, Dumitru and Bengio, Y. and Courville, Aaron and Vincent, Pascal}, +year = {2009}, +month = {01}, +title = {Visualizing Higher-Layer Features of a Deep Network}, +journal = {Technical Report, Univeristé de Montréal} } -@misc{kim2018interpretability, - title={Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)}, - author={Been Kim and Martin Wattenberg and Justin Gilmer and Carrie Cai and James Wexler and Fernanda Viegas and Rory Sayres}, - year={2018}, - eprint={1711.11279}, - archivePrefix={arXiv}, - primaryClass={stat.ML} +@InProceedings{kim2017interpretability, + title={Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors ({TCAV})}, + author={Kim, Been and Wattenberg, Martin and Gilmer, Justin and Cai, Carrie and Wexler, James and Viegas, Fernanda and sayres, Rory}, + booktitle={Proceedings of the 35th International Conference on Machine Learning}, + pages={2668--2677}, + year={2018}, + volume={80}, + publisher={PMLR} +} + + +@InProceedings{wang2022tbnet, + title={Tower Bridge Net (TB-Net): Bidirectional Knowledge Graph Aware Embedding Propagation for Explainable Recommender System}, + author={Wang, Shendi and Li, Haoyang and Cao, Caleb Chen and Li, Xiao-Hui and Ng, Ngai Fai and Liu, Jianxin and Xue, Xun and Song, Hu and Li, Jinyu, and Gu, Guangye and Chen, Lei}, + booktitle={Proceedings of 38th IEEE International Conference on Data Engineering}, + pages={Accepted}, + year={2022}, + publisher={IEEE} } @article{riedl2019human, - title={Human-centered artificial intelligence and machine learning}, - author={Riedl, Mark O.}, - journal={Human Behavior and Emerging Technologies}, - volume={1}, - number={1}, - pages={33--36}, - year={2019}, - publisher={Wiley Online Library} - + title={Human-centered artificial intelligence and machine learning}, + author={Riedl, Mark O.}, + journal={Human Behavior and Emerging Technologies}, + volume={1}, + number={1}, + pages={33--36}, + year={2019}, + publisher={Wiley Online Library} } +@inproceedings{ribeiro2016should, + title={" Why should i trust you?" Explaining the predictions of any classifier}, + author={Ribeiro, Marco Tulio and Singh, Sameer and Guestrin, Carlos}, + booktitle={Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining}, + pages={1135--1144}, + year={2016} +} + +@article{frosst2017distilling, + title={Distilling a neural network into a soft decision tree}, + author={Frosst, Nicholas and Hinton, Geoffrey}, + journal={arXiv preprint arXiv:1711.09784}, + year={2017} +} + +@article{lundberg2017unified, + title={A unified approach to interpreting model predictions}, + author={Lundberg, Scott M and Lee, Su-In}, + journal={Advances in neural information processing systems}, + volume={30}, + year={2017} +} + +@article{chen2019looks, + title={This looks like that: deep learning for interpretable image recognition}, + author={Chen, Chaofan and Li, Oscar and Tao, Daniel and Barnett, Alina and Rudin, Cynthia and Su, Jonathan K}, + journal={Advances in neural information processing systems}, + volume={32}, + year={2019} +} + +@inproceedings{zhang2019interpreting, + title={Interpreting cnns via decision trees}, + author={Zhang, Quanshi and Yang, Yu and Ma, Haotian and Wu, Ying Nian}, + booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition}, + pages={6261--6270}, + year={2019} +} + +@inproceedings{zhang2018interpretable, + title={Interpretable convolutional neural networks}, + author={Zhang, Quanshi and Wu, Ying Nian and Zhu, Song-Chun}, + booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition}, + pages={8827--8836}, + year={2018} +} + +@inproceedings{nauta2021neural, + title={Neural prototype trees for interpretable fine-grained image recognition}, + author={Nauta, Meike and van Bree, Ron and Seifert, Christin}, + booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, + pages={14933--14943}, + year={2021} +} + +@inproceedings{selvaraju2017grad, + title={Grad-cam: Visual explanations from deep networks via gradient-based localization}, + author={Selvaraju, Ramprasaath R and Cogswell, Michael and Das, Abhishek and Vedantam, Ramakrishna and Parikh, Devi and Batra, Dhruv}, + booktitle={Proceedings of the IEEE international conference on computer vision}, + pages={618--626}, + year={2017} +} + +@inproceedings{zeiler2014visualizing, + title={Visualizing and understanding convolutional networks}, + author={Zeiler, Matthew D and Fergus, Rob}, + booktitle={European conference on computer vision}, + pages={818--833}, + year={2014}, + organization={Springer} +} + +@article{petsiuk2018rise, + title={Rise: Randomized input sampling for explanation of black-box models}, + author={Petsiuk, Vitali and Das, Abir and Saenko, Kate}, + journal={arXiv preprint arXiv:1806.07421}, + year={2018} +} + +@inproceedings{wang2020score, + title={Score-CAM: Score-weighted visual explanations for convolutional neural networks}, + author={Wang, Haofan and Wang, Zifan and Du, Mengnan and Yang, Fan and Zhang, Zijian and Ding, Sirui and Mardziel, Piotr and Hu, Xia}, + booktitle={Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops}, + pages={24--25}, + year={2020} +} + + @inproceedings{10.1145/2988450.2988454, -author = {Cheng, Heng-Tze and Koc, Levent and Harmsen, Jeremiah and Shaked, Tal and Chandra, Tushar and Aradhye, Hrishi and Anderson, Glen and Corrado, Greg and Chai, Wei and Ispir, Mustafa and Anil, Rohan and Haque, Zakaria and Hong, Lichan and Jain, Vihan and Liu, Xiaobing and Shah, Hemal}, -title = {Wide & Deep Learning for Recommender Systems}, -year = {2016}, -isbn = {9781450347952}, -publisher = {Association for Computing Machinery}, -address = {New York, NY, USA}, -url = {https://doi.org/10.1145/2988450.2988454}, -doi = {10.1145/2988450.2988454}, -abstract = {Generalized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort. With less feature engineering, deep neural networks can generalize better to unseen feature combinations through low-dimensional dense embeddings learned for the sparse features. However, deep neural networks with embeddings can over-generalize and recommend less relevant items when the user-item interactions are sparse and high-rank. In this paper, we present Wide & Deep learning---jointly trained wide linear models and deep neural networks---to combine the benefits of memorization and generalization for recommender systems. We productionized and evaluated the system on Google Play, a commercial mobile app store with over one billion active users and over one million apps. Online experiment results show that Wide & Deep significantly increased app acquisitions compared with wide-only and deep-only models. We have also open-sourced our implementation in TensorFlow.}, -booktitle = {Proceedings of the 1st Workshop on Deep Learning for Recommender Systems}, -pages = {7-10}, -numpages = {4}, -keywords = {Recommender Systems, Wide & Deep Learning}, -location = {Boston, MA, USA}, -series = {DLRS 2016} -} \ No newline at end of file + author = {Cheng, Heng-Tze and Koc, Levent and Harmsen, Jeremiah and Shaked, Tal and Chandra, Tushar and Aradhye, Hrishi and Anderson, Glen and Corrado, Greg and Chai, Wei and Ispir, Mustafa and Anil, Rohan and Haque, Zakaria and Hong, Lichan and Jain, Vihan and Liu, Xiaobing and Shah, Hemal}, + title = {Wide & Deep Learning for Recommender Systems}, + year = {2016}, + isbn = {9781450347952}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + url = {https://doi.org/10.1145/2988450.2988454}, + doi = {10.1145/2988450.2988454}, + abstract = {Generalized linear models with nonlinear feature transformations are widely used for large-scale regression and classification problems with sparse inputs. Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort. With less feature engineering, deep neural networks can generalize better to unseen feature combinations through low-dimensional dense embeddings learned for the sparse features. However, deep neural networks with embeddings can over-generalize and recommend less relevant items when the user-item interactions are sparse and high-rank. In this paper, we present Wide & Deep learning---jointly trained wide linear models and deep neural networks---to combine the benefits of memorization and generalization for recommender systems. We productionized and evaluated the system on Google Play, a commercial mobile app store with over one billion active users and over one million apps. Online experiment results show that Wide & Deep significantly increased app acquisitions compared with wide-only and deep-only models. We have also open-sourced our implementation in TensorFlow.}, + booktitle = {Proceedings of the 1st Workshop on Deep Learning for Recommender Systems}, + pages = {7-10}, + numpages = {4}, + keywords = {Recommender Systems, Wide & Deep Learning}, + location = {Boston, MA, USA}, + series = {DLRS 2016} +}