diff --git a/chapter_reinforcement_learning/marl_sys.md b/chapter_reinforcement_learning/marl_sys.md index 92746d8..7c45664 100644 --- a/chapter_reinforcement_learning/marl_sys.md +++ b/chapter_reinforcement_learning/marl_sys.md @@ -1,6 +1,6 @@ ## 多智能体强化学习系统 -上述的简单例子只是为了帮助读者理解强化学习在多智能体问题里的角色，而如今前沿的多智能体强化学习算法已经能够解决相当大规模的复杂多智能体问题，如星际争霸（StarCraft II）、Dota 2等游戏，已相继被DeepMind、OpenAI等公司所研究的智能体AlphaStar~\citep{vinyals2019grandmaster}和OpenAI Five~\citep{berner2019dota}攻克，达到超越人类顶级玩家的水平。国内公司如腾讯、启元世界等也提出了星际争霸游戏的多智能体强化学习解决方案TStarBot-X~\citep{han2020tstarbot}和SCC~\citep{wang2021scc}。对于这类高度复杂的游戏环境，整个训练过程对分布式计算系统的要求更高，而整个训练过程可能需要分为多个阶段。以AlphaStar为例，它训练的智能体采用了监督学习与强化学习结合的方式。在训练早期，往往先采用大量的人类专业玩家标定数据进行有监督的学习，从而使智能体快速获得较好的能力，随后，训练会切换到强化学习过程，使用前面介绍的虚构自学习的算法进行训练，即自我博弈。为了得到一个表现最好的智能体，算法需要充分探索整个策略空间，从而在训练中不止对一个策略进行训练，而是对一个策略集群（League）进行训练，并通过类似演化算法的方式对策略集群进行筛选，得到大量策略中表现最好的策略。如图:numref:`ch12/ch12-marl_train`所示，在训练过程中每个智能体往往需要和其他智能体以及剥削者（Exploiter）进行博弈，剥削者是专门针对某一个智能体策略的最佳对手策略，与之对抗可以提高策略自身的防剥削能力。通过对大量智能体策略进行训练并筛选的这类方法称为集群式训练（Population-based Training/League Training），是一种通过分布式训练提高策略种群多样性进而提升模型表现的方式。可见，在实践中这类方法自然需要分布式系统支持，来实现多个智能体的训练和相互博弈，这很好地体现了多智能体强化学习对分布式计算的依赖性。 +上述的简单例子只是为了帮助读者理解强化学习在多智能体问题里的角色，而如今前沿的多智能体强化学习算法已经能够解决相当大规模的复杂多智能体问题，如星际争霸（StarCraft II）、Dota 2等游戏，已相继被DeepMind、OpenAI等公司所研究的智能体AlphaStar:cite:`vinyals2019grandmaster`和OpenAI Five:cite:`berner2019dota`攻克，达到超越人类顶级玩家的水平。国内公司如腾讯、启元世界等也提出了星际争霸游戏的多智能体强化学习解决方案TStarBot-X:cite:`han2020tstarbot`和SCC:cite:`wang2021scc`。对于这类高度复杂的游戏环境，整个训练过程对分布式计算系统的要求更高，而整个训练过程可能需要分为多个阶段。以AlphaStar为例，它训练的智能体采用了监督学习与强化学习结合的方式。在训练早期，往往先采用大量的人类专业玩家标定数据进行有监督的学习，从而使智能体快速获得较好的能力，随后，训练会切换到强化学习过程，使用前面介绍的虚构自学习的算法进行训练，即自我博弈。为了得到一个表现最好的智能体，算法需要充分探索整个策略空间，从而在训练中不止对一个策略进行训练，而是对一个策略集群（League）进行训练，并通过类似演化算法的方式对策略集群进行筛选，得到大量策略中表现最好的策略。如图:numref:`ch12/ch12-marl_train`所示，在训练过程中每个智能体往往需要和其他智能体以及剥削者（Exploiter）进行博弈，剥削者是专门针对某一个智能体策略的最佳对手策略，与之对抗可以提高策略自身的防剥削能力。通过对大量智能体策略进行训练并筛选的这类方法称为集群式训练（Population-based Training/League Training），是一种通过分布式训练提高策略种群多样性进而提升模型表现的方式。可见，在实践中这类方法自然需要分布式系统支持，来实现多个智能体的训练和相互博弈，这很好地体现了多智能体强化学习对分布式计算的依赖性。 我们将对构建多智能体强化学习系统中的困难分为以下几点进行讨论： @@ -10,4 +10,4 @@ * **算法的异构**：从前面介绍的几个简单的多智能体算法，如自学习、虚构自学习等可以看出，多智能体算法有时由许多轮单智能体强化学习过程组成。而对不同的游戏类型，算法的类型也不相同。比如，对合作型游戏，许多算法是基于奖励分配（Credit Assignment）的思想，如何将多个智能体获得的共同奖励合理分配给单个智能体是这类算法的核心。而这里面按照具体算法执行方式，也可以分为集成训练统一执行的（Centralized Training Centralized Execution）、集成训练分别执行的（Centralized Training Decentralized Execution）、分别训练并分别执行（Decentralized Training Decentralized Execution）的几类，来描述不同智能体训练过程和执行过程的统一性。对于竞争型游戏，往往采用各种计算纳什均衡的近似方法，如前面提到的虚构自学习、Double Oracle、Mirror Descent等等，将获取单个最优策略的单智能体强化学习过程看做一个“动作”，而对这些“动作”组成的元问题上进行纳什均衡近似。现有的算法在类似问题上有很大的差异性，使得构建一个统一的多智能体强化学习系统比较困难。 -* **学习方法组合**：在前面提到的AlphaStar~\citep{vinyals2019grandmaster}等工作中，多智能体系统中优化得到一个好的策略往往不只需要强化学习算法，还需要其他学习方法如模仿学习等的辅助。比如从一些顶级人类玩家的游戏记录中形成有标签的训练样本，来预训练智能体。由于这些大规模游戏的复杂性，这往往是一个在训练前期快速提升智能体表现的有效方式。而对于整个学习系统而言，这就需要对不同学习范式进行结合，如合理地在模仿学习和强化学习之间进行切换等。这也使得大规模多智能体系统不单一是构建强化学习系统的问题，而需要许多其他学习机制和协调机制的配合实现。 \ No newline at end of file +* **学习方法组合**：在前面提到的AlphaStar:cite:`vinyals2019grandmaster`等工作中，多智能体系统中优化得到一个好的策略往往不只需要强化学习算法，还需要其他学习方法如模仿学习等的辅助。比如从一些顶级人类玩家的游戏记录中形成有标签的训练样本，来预训练智能体。由于这些大规模游戏的复杂性，这往往是一个在训练前期快速提升智能体表现的有效方式。而对于整个学习系统而言，这就需要对不同学习范式进行结合，如合理地在模仿学习和强化学习之间进行切换等。这也使得大规模多智能体系统不单一是构建强化学习系统的问题，而需要许多其他学习机制和协调机制的配合实现。 \ No newline at end of file diff --git a/chapter_reinforcement_learning/single_node_rl.md b/chapter_reinforcement_learning/single_node_rl.md index c08c3de..280c4d2 100644 --- a/chapter_reinforcement_learning/single_node_rl.md +++ b/chapter_reinforcement_learning/single_node_rl.md @@ -3,11 +3,11 @@ 前面介绍了强化学习的基本知识和在系统层面的一般需求，这里我们介绍常见的单智能体强化学习系统中较为简单的一类，即单节点强化学习系统。这里，我们按照是否对模型训练和更新进行并行处理，将强化学习系统分为单节点和分布式强化学习系统。其中，单节点强化学习系统可以理解为只实例化一个类对象作为智能体，与环境交互进行采样和利用所采得的样本进行更新的过程分别视为这个类内的不同函数。除此之外的更为复杂的强化学习框架都可视为分布式强化学习系统。分布式强化学习系统的具体形式有很多，这也往往依赖于所实现的算法。从最简单的情况考虑，假设我们仍在同一个计算单元上实现算法，但是将强化学习的采样过程和更新过程实现为两个并行的进程，甚至各自实现为多个进程，以满足不同计算资源间的平衡。这时就需要进程间通信来协调采样和更新过程，这是一个最基础的分布式强化学习框架。更为复杂的情况是，整个算法的运行在多个计算设备上进行（如一个多机的计算集群），智能体的函数可能需要跨机跨进程间的通信来实现。对于多智能体系统，还需要同时对多个智能体的模型进行更新，则需要更为复杂的计算系统设计。我们将逐步介绍这些不同的系统内的实现机制。 我们先对单节点强化学习系统进行介绍。 -在这里，我们以RLzoo\citep{ding2020efficient}为例，讲解一个单节点强化学习系统构建所需要的基本模块。如图:numref:`ch12/ch12-rlzoo`所示，是RLzoo中采用的一个典型的单节点强化学习系统，它包括几个基本的组成部分：神经网络、适配器、策略网络和价值网络、环境实例、模型学习器、经验回放缓存（Experience Replay Buffer）等。我们先对前三个，神经网络、适配器、策略网络和价值网络进行介绍。神经网络即一般深度学习中的神经网络，用于实现基于数据的函数拟合，我们在图中简单列出常见的三类神经网络：全连接网络，卷积网络和循环网络。策略网络和价值网络是一般深度强化学习的常见组成部分，策略网络即一个由深度神经网络参数化的策略表示，而价值网络为神经网络表示的状态价值（State-Value）或状态-动作价值（State-Action Value）函数。这里我们不妨称前三类神经网络为一般神经网络，策略网络和价值网络为强化学习特定网络，前者往往是后者的重要组成部分。在RLzoo中，适配器则是为实现强化学习特定网络而选配一般神经网络的功能模块。首先，根据不同的观察量类型，强化学习智能体所用的神经网络头部会有不同的结构，这一选择可以由一个基于观察量的适配器来实现；其次，根据所采用的强化学习算法类型，相应的策略网络尾部需要有不同的输出类型，包括确定性策略和随机性策略，RLzoo中使用一个策略适配器来进行选择；最后，根据不同的动作输出，如离散型、连续型、类别型等，需要使用一个动作适配器来选择。图:numref:`fig:ch12/ch12-rlzoo`中我们统称这三个不类型的适配器为适配器。介绍完这些，我们已经有了可用的策略网络和价值网络，这构成了强化学习智能体核心学习模块。除此之外，还需要一个学习器（Learner）来更新这些学习模块，更新的规则就是强化学习算法给出的损失函数。而要想实现学习模块的更新，最重要的是输入的学习数据，即智能体跟环境交互过程中所采集的样本。对于**离线**（Off-Policy）强化学习，这些样本通常被存储于一个称为经验回放缓存的地方，学习器在需要更新模型时从该缓存中采得一些样本来进行更新。这里说到的离线强化学习是强化学习算法中的一类，强化学习算法可以分为**在线**（On-Policy）强化学习和离线强化学习两类，按照某个特定判据。这个判据是，用于更新的模型和用于采样的模型是否为同一个，如果是，则称在线强化学习算法，否则为离线强化学习算法。因而，离线强化学习通常允许与环境交互的策略采集的样本被存储于一个较大的缓存内，从而允许在许久之后再从这个缓存中抽取样本对模型进行更新。而对于在线强化学习，这个“缓存”有时其实也是存在的，只不过它所存储的是非常近期内采集的数据，从而被更新模型和用于采样的模型可以近似认为是同一个。从而，这里我们简单表示RLzoo的强化学习系统统一包括这个经验回放缓存模块。有了以上策略和价值网络、经验回放缓存、适配器、学习器，我们就得到了RLzoo中一个单节点的强化学习智能体，将这个智能体与环境实例交互，并采集数据进行模型更新，我们就得到了一个完整的单节点强化学习系统。这里的环境实例化我们允许多个环境并行采样。 +在这里，我们以RLzoo:cite:`ding2020efficient`为例，讲解一个单节点强化学习系统构建所需要的基本模块。如图:numref:`ch12/ch12-rlzoo`所示，是RLzoo中采用的一个典型的单节点强化学习系统，它包括几个基本的组成部分：神经网络、适配器、策略网络和价值网络、环境实例、模型学习器、经验回放缓存（Experience Replay Buffer）等。我们先对前三个，神经网络、适配器、策略网络和价值网络进行介绍。神经网络即一般深度学习中的神经网络，用于实现基于数据的函数拟合，我们在图中简单列出常见的三类神经网络：全连接网络，卷积网络和循环网络。策略网络和价值网络是一般深度强化学习的常见组成部分，策略网络即一个由深度神经网络参数化的策略表示，而价值网络为神经网络表示的状态价值（State-Value）或状态-动作价值（State-Action Value）函数。这里我们不妨称前三类神经网络为一般神经网络，策略网络和价值网络为强化学习特定网络，前者往往是后者的重要组成部分。在RLzoo中，适配器则是为实现强化学习特定网络而选配一般神经网络的功能模块。首先，根据不同的观察量类型，强化学习智能体所用的神经网络头部会有不同的结构，这一选择可以由一个基于观察量的适配器来实现；其次，根据所采用的强化学习算法类型，相应的策略网络尾部需要有不同的输出类型，包括确定性策略和随机性策略，RLzoo中使用一个策略适配器来进行选择；最后，根据不同的动作输出，如离散型、连续型、类别型等，需要使用一个动作适配器来选择。图:numref:`fig:ch12/ch12-rlzoo`中我们统称这三个不类型的适配器为适配器。介绍完这些，我们已经有了可用的策略网络和价值网络，这构成了强化学习智能体核心学习模块。除此之外，还需要一个学习器（Learner）来更新这些学习模块，更新的规则就是强化学习算法给出的损失函数。而要想实现学习模块的更新，最重要的是输入的学习数据，即智能体跟环境交互过程中所采集的样本。对于**离线**（Off-Policy）强化学习，这些样本通常被存储于一个称为经验回放缓存的地方，学习器在需要更新模型时从该缓存中采得一些样本来进行更新。这里说到的离线强化学习是强化学习算法中的一类，强化学习算法可以分为**在线**（On-Policy）强化学习和离线强化学习两类，按照某个特定判据。这个判据是，用于更新的模型和用于采样的模型是否为同一个，如果是，则称在线强化学习算法，否则为离线强化学习算法。因而，离线强化学习通常允许与环境交互的策略采集的样本被存储于一个较大的缓存内，从而允许在许久之后再从这个缓存中抽取样本对模型进行更新。而对于在线强化学习，这个“缓存”有时其实也是存在的，只不过它所存储的是非常近期内采集的数据，从而被更新模型和用于采样的模型可以近似认为是同一个。从而，这里我们简单表示RLzoo的强化学习系统统一包括这个经验回放缓存模块。有了以上策略和价值网络、经验回放缓存、适配器、学习器，我们就得到了RLzoo中一个单节点的强化学习智能体，将这个智能体与环境实例交互，并采集数据进行模型更新，我们就得到了一个完整的单节点强化学习系统。这里的环境实例化我们允许多个环境并行采样。 [loading-ag-175](../img/ch12/ch12-rlzoo.png) :width:`800px` :label:`ch12/ch12-rlzoo` -近来研究人员发现，强化学习算法领域的发展瓶颈，可能不仅在于算法本身，而在于智能体采集数据的模拟器的模拟速度。Issac Gym\citep{makoviychuk2021isaac}是Nvidia公司于2021年推出的基于GPU的模拟引擎，在单GPU上实现2-3倍于之前基于CPU的模拟器的运行速度。关于GPU上运行加速我们已经在章节5中有所介绍。之所以GPU模拟能够对强化学习任务实现显著的加速效果，除了GPU本身多核心的并行运算能力之外，还在于这省却了CPU与GPU之间的数据传输和通信时间。传统的强化学习环境，如OpenAI Gym（这是一个常用的强化学习基准测试环境）等，都是基于CPU进行的模拟计算，而深度学习方法的神经网络训练通常是在GPU或TPU上进行的。从智能体与CPU上实例化的模拟环境交互过程所收集的数据样本，通常先暂时以CPU的数据格式存储，在使用的时候被转移到GPU上成为具有GPU数据类型的数据（如使用PyTorch时可通过tensor.to(device)的函数实现，只需将device设为“cuda”即可将一个类型为troch.Tensor的tensor转移到GPU上），然后来进行模型训练。同时，由于模型参数是以GPU上数据的类型存储的，调用模型进行前向传递的过程中也需要先将输入数据从CPU转移到GPU上，并且可能需要将模型输出的GPU数据再转移回CPU类型。这一系列冗余的数据转换操作都会显著增长模型学习的时间，并且也增加了算法实际使用过程中的工程量。Isaac Gym模拟器的设计从底层上解决了这一困难，由于模拟器和模型双双实现在GPU上，他们之间的数据通信不再需要通过CPU来实现，从而绕过了CPU与GPU数据双向传输这一问题，实现了对强化学习任务中模拟过程的特定加速。 \ No newline at end of file +近来研究人员发现，强化学习算法领域的发展瓶颈，可能不仅在于算法本身，而在于智能体采集数据的模拟器的模拟速度。Issac Gym:cite:`makoviychuk2021isaac`是Nvidia公司于2021年推出的基于GPU的模拟引擎，在单GPU上实现2-3倍于之前基于CPU的模拟器的运行速度。关于GPU上运行加速我们已经在章节5中有所介绍。之所以GPU模拟能够对强化学习任务实现显著的加速效果，除了GPU本身多核心的并行运算能力之外，还在于这省却了CPU与GPU之间的数据传输和通信时间。传统的强化学习环境，如OpenAI Gym（这是一个常用的强化学习基准测试环境）等，都是基于CPU进行的模拟计算，而深度学习方法的神经网络训练通常是在GPU或TPU上进行的。从智能体与CPU上实例化的模拟环境交互过程所收集的数据样本，通常先暂时以CPU的数据格式存储，在使用的时候被转移到GPU上成为具有GPU数据类型的数据（如使用PyTorch时可通过tensor.to(device)的函数实现，只需将device设为“cuda”即可将一个类型为troch.Tensor的tensor转移到GPU上），然后来进行模型训练。同时，由于模型参数是以GPU上数据的类型存储的，调用模型进行前向传递的过程中也需要先将输入数据从CPU转移到GPU上，并且可能需要将模型输出的GPU数据再转移回CPU类型。这一系列冗余的数据转换操作都会显著增长模型学习的时间，并且也增加了算法实际使用过程中的工程量。Isaac Gym模拟器的设计从底层上解决了这一困难，由于模拟器和模型双双实现在GPU上，他们之间的数据通信不再需要通过CPU来实现，从而绕过了CPU与GPU数据双向传输这一问题，实现了对强化学习任务中模拟过程的特定加速。 \ No newline at end of file diff --git a/mlsys.bib b/mlsys.bib index fd33fc6..f5ed2c7 100644 --- a/mlsys.bib +++ b/mlsys.bib @@ -101,3 +101,9574 @@ pages={2121--2159}, year={2011} } + + +@inproceedings{vaswani2017attention, + title={Attention is all you need}, + author={Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N and Kaiser, {\L}ukasz and Polosukhin, Illia}, + booktitle={Advances in Neural Information Processing Systems}, + pages={5998--6008}, + year={2017} +} + +@article{attentionTS, + title={Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer}, + author={Zagoruyko, Sergey and Komodakis, Nikos}, + journal={arXiv preprint arXiv:1612.03928}, + year={2016} +} + +@inproceedings{bagherinezhad2017lcnn, + title={Lcnn: Lookup-based convolutional neural network}, + author={Bagherinezhad, Hessam and Rastegari, Mohammad and Farhadi, Ali}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={7120--7129}, + year={2017} +} + +@article{Distill, + title={Distilling the knowledge in a neural network}, + author={Hinton, Geoffrey and Vinyals, Oriol and Dean, Jeff}, + journal={arXiv preprint arXiv:1503.02531}, + year={2015} +} +@article{FitNet, + title={Fitnets: Hints for thin deep nets}, + author={Romero, Adriana and Ballas, Nicolas and Kahou, Samira Ebrahimi and Chassang, Antoine and Gatta, Carlo and Bengio, Yoshua}, + journal={arXiv preprint arXiv:1412.6550}, + year={2014} +} + +@article{han2015deep, + title={Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding}, + author={Han, Song and Mao, Huizi and Dally, William J}, + journal={arXiv preprint arXiv:1510.00149}, + year={2015} +} + +%% ML history +@article{hornik1989multilayer, + title={Multilayer feedforward networks are universal approximators}, + author={Hornik, Kurt and Stinchcombe, Maxwell and White, Halbert}, + journal={Neural networks}, + volume={2}, + number={5}, + pages={359--366}, + year={1989}, + publisher={Elsevier} +} + +@inproceedings{samuel1959ml, + title={Some Studies in Machine Learning Using the Game of Checkers}, + author={Samuel, Arthur}, + booktitle={IBM Journal of Research and Development}, + year={1959} +} + +@article{mcculloch1943logical, + title={A logical calculus of the ideas immanent in nervous activity}, + author={McCulloch, Warren S and Pitts, Walter}, + journal={The bulletin of mathematical biophysics}, + volume={5}, + number={4}, + pages={115--133}, + year={1943}, + publisher={Springer} +} + +@inproceedings{mai2020kungfu, + title={Kungfu: Making training in distributed machine learning adaptive}, + author={Mai, Luo and Li, Guo and Wagenl{\"a}nder, Marcel and Fertakis, Konstantinos and Brabete, Andrei-Octavian and Pietzuch, Peter}, + booktitle={14th $\{$USENIX$\}$ Symposium on Operating Systems Design and Implementation ($\{$OSDI$\}$ 20)}, + pages={937--954}, + year={2020} +} + +%%% DL Basic +@inproceedings{zhang2015textnips, + title={Character-level convolutional networks for text classification}, + author={Zhang, Xiang and Zhao, Junbo and LeCun, Yann}, + booktitle={Advances in Neural Information Processing Systems}, + pages={649--657}, + year={2015} +} + +@inproceedings{devlin-etal-2019-bert, + title = "{BERT}: Pre-training of Deep Bidirectional Transformers for Language Understanding", + author = "Devlin, Jacob and + Chang, Ming-Wei and + Lee, Kenton and + Toutanova, Kristina", + booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)", + month = jun, + year = "2019", + address = "Minneapolis, Minnesota", + publisher = "Association for Computational Linguistics", + url = "https://www.aclweb.org/anthology/N19-1423", + doi = "10.18653/v1/N19-1423", + pages = "4171--4186", + abstract = "We introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modifications. BERT is conceptually simple and empirically powerful. It obtains new state-of-the-art results on eleven natural language processing tasks, including pushing the GLUE score to 80.5 (7.7 point absolute improvement), MultiNLI accuracy to 86.7{\%} (4.6{\%} absolute improvement), SQuAD v1.1 question answering Test F1 to 93.2 (1.5 point absolute improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).", +} + +@inproceedings{yang2019xlnet, + title = {XLNet: Generalized Autoregressive Pretraining for Language Understanding}, + author = {Yang, Zhilin and Dai, Zihang and Yang, Yiming and Carbonell, Jaime and Salakhutdinov, Russ R and Le, Quoc V}, + booktitle = {Advances in Neural Information Processing Systems}, + pages = {5754--5764}, + year = {2019}, +} + +@article{radford2019language, + title={Language models are unsupervised multitask learners}, + author={Radford, Alec and Wu, Jeffrey and Child, Rewon and Luan, David and Amodei, Dario and Sutskever, Ilya}, + journal={OpenAI Blog}, + volume={1}, + number={8}, + year={2019} +} + +@article{ioffe2015batch, + title={Batch normalization: Accelerating deep network training by reducing internal covariate shift}, + author={Ioffe, Sergey and Szegedy, Christian}, + journal={arXiv preprint arXiv:1502.03167}, + year={2015} +} + +@article{cybenko1989approximation, + title={Approximation by superpositions of a sigmoidal function}, + author={Cybenko, George}, + journal={Mathematics of control, signals and systems}, + volume={2}, + number={4}, + pages={303--314}, + year={1989}, + publisher={Springer} +} + +@article{bubeck2012regret, + title={Regret analysis of stochastic and nonstochastic multi-armed bandit problems}, + author={Bubeck, S{\'e}bastien and Cesa-Bianchi, Nicolo and others}, + journal={Foundations and Trends{\textregistered} in Machine Learning}, + volume={5}, + number={1}, + pages={1--122}, + year={2012}, + publisher={Now Publishers, Inc.} +} + +@book{bishop2006pattern, + title={Pattern recognition and machine learning}, + author={Bishop, Christopher M}, + year={2006}, + publisher={springer} +} + +% already in this bibtext +% @book{Goodfellow-et-al-2016, +% title={Deep Learning}, +% author={Ian Goodfellow and Yoshua Bengio and Aaron Courville}, +% publisher={MIT Press}, +% note={\url{http://www.deeplearningbook.org}}, +% year={2016} +% } + +@inproceedings{zoph2018learning, + title={Learning transferable architectures for scalable image recognition}, + author={Zoph, Barret and Vasudevan, Vijay and Shlens, Jonathon and Le, Quoc V}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={8697--8710}, + year={2018} +} + +@article{zoph2016neural, + title={Neural architecture search with reinforcement learning}, + author={Zoph, Barret and Le, Quoc V}, + journal={arXiv preprint arXiv:1611.01578}, + year={2016} +} + +@inproceedings{krizhevsky2012imagenet, + title={Imagenet classification with deep convolutional neural networks}, + author={Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1097--1105}, + year={2012} +} + +@article{lecun1989backpropagation, + title={Backpropagation applied to handwritten zip code recognition}, + author={LeCun, Yann and Boser, Bernhard and Denker, John S and Henderson, Donnie and Howard, Richard E and Hubbard, Wayne and Jackel, Lawrence D}, + journal={Neural computation}, + volume={1}, + number={4}, + pages={541--551}, + year={1989}, + publisher={MIT Press} +} + +@article{lecun2015deep, + title={Deep learning}, + author={LeCun, Yann and Bengio, Yoshua and Hinton, Geoffrey}, + journal={Nature}, + volume={521}, + number={7553}, + pages={436}, + year={2015}, + publisher={Nature Publishing Group} +} + +@article{lecun1998gradient, + title={Gradient-based learning applied to document recognition}, + author={LeCun, Yann and Bottou, L{\'e}on and Bengio, Yoshua and Haffner, Patrick and others}, + journal={Proceedings of the IEEE}, + volume={86}, + number={11}, + pages={2278--2324}, + year={1998}, + publisher={Taipei, Taiwan} +} + +@article{rumelhart1986learning, + title={Learning representations by back-propagating errors}, + author={Rumelhart, David E and Hinton, Geoffrey E and Williams, Ronald J}, + journal={Nature}, + volume={323}, + number={6088}, + pages={533}, + year={1986}, + publisher={Nature Publishing Group} +} + +@misc{olah_2015, + title={Understanding LSTM Networks}, url={https://colah.github.io/posts/2015-08-Understanding-LSTMs/}, + author={Olah, Christopher}, + year={2015}, + month={Aug} +} + +@inproceedings{glorot2011relu, + title={Deep sparse rectifier neural networks}, + author={Glorot, Xavier and Bordes, Antoine and Bengio, Yoshua}, + booktitle={Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS)}, + pages={315--323}, + year={2011} +} +@article{lennie2003cost, + title={The cost of cortical computation}, + author={Lennie, Peter}, + journal={Current biology}, + volume={13}, + number={6}, + pages={493--497}, + year={2003}, + publisher={Elsevier} +} + +@inproceedings{xu2015leakyrelu, + title={Empirical evaluation of rectified activations in convolutional network}, + author={Xu, Bing and Wang, Naiyan and Chen, Tianqi and Li, Mu}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML) Workshop}, + year={2015} +} +@inproceedings{he2015prelu, + title={Delving deep into rectifiers: Surpassing human-level performance on imagenet classification}, + author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian}, + booktitle={Proceedings of the IEEE international conference on computer vision}, + pages={1026--1034}, + year={2015} +} +@article{srivastava2014dropout, + title={Dropout: A simple way to prevent neural networks from overfitting}, + author={Srivastava, Nitish and Hinton, Geoffrey and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={15}, + number={1}, + pages={1929--1958}, + year={2014}, +} +@article{HintonDropout2012, + title={Improving neural networks by preventing co-adaptation of feature detectors}, + author={Hinton, Geoffrey E and Srivastava, Nitish and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan R}, + journal={arXiv preprint arXiv:1207.0580}, + year={2012} +} + +@inproceedings{auer1995gambling, + title={Gambling in a rigged casino: The adversarial multi-armed bandit problem}, + author={Auer, Peter and Cesa-Bianchi, Nicolo and Freund, Yoav and Schapire, Robert E}, + booktitle={Proceedings of IEEE 36th Annual Foundations of Computer Science}, + pages={322--331}, + year={1995}, + organization={IEEE} +} + +@inproceedings{gal2016dropoutbayesian, + title={Dropout as a Bayesian approximation: Representing model uncertainty in deep learning}, + author={Gal, Yarin and Ghahramani, Zoubin}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + pages={1050--1059}, + year={2016} +} +@inproceedings{hara2017dropoutensemble, + title={Analysis of dropout learning regarded as ensemble learning}, + author={Hara, Kazuyuki and Saitoh, Daisuke and Shouno, Hayaru}, + booktitle={Proceedings of the International Conference on Artificial Neural Networks (ICANN)}, + pages={72--79}, + year={2016}, + organization={Springer} +} +@inproceedings{WanDropconnect2013, + title={Regularization of neural networks using dropconnect}, + author={Wan, Li and Zeiler, Matthew and Zhang, Sixin and Le Cun, Yann and Fergus, Rob}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + pages={1058--1066}, + year={2013} +} +@article{UncertaintyPhD, + title={Uncertainty in deep learning}, + author={Gal, Yarin}, + journal={University of Cambridge}, + year={2016} +} +@article{RandomGaussianWeights, + title={Deep neural networks with random gaussian weights: A universal classification strategy?}, + author={Giryes, Raja and Sapiro, Guillermo and Bronstein, Alexander M}, + journal={IEEE Transaction of Signal Processing}, + volume={64}, + number={13}, + pages={3444--3457}, + year={2016} +} +@inproceedings{BlundellGaussianNN2015, + title={Weight uncertainty in neural networks}, + author={Blundell, Charles and Cornebise, Julien and Kavukcuoglu, Koray and Wierstra, Daan}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2015} +} +@article{goodfellow2013maxout, + title={Maxout networks}, + author={Goodfellow, Ian J and Warde-Farley, David and Mirza, Mehdi and Courville, Aaron and Bengio, Yoshua}, + journal={The Journal of Machine Learning Research (JMLR)}, + year={2013} +} + +@inproceedings{maas2011sentiment, + author = {Maas, Andrew L. and Daly, Raymond E. and Pham, Peter T. and Huang, Dan and Ng, Andrew Y. and Potts, Christopher}, + title = {Learning Word Vectors for Sentiment Analysis}, + booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1}, + series = {HLT '11}, + year = {2011}, + isbn = {978-1-932432-87-9}, + location = {Portland, Oregon}, + pages = {142--150}, + numpages = {9}, + url = {http://dl.acm.org/citation.cfm?id=2002472.2002491}, + acmid = {2002491}, + publisher = {Association for Computational Linguistics}, + address = {Stroudsburg, PA, USA}, +} + +@inproceedings{wavenet2016, +title = {{WaveNet}: A Generative Model for Raw Audio}, +author = {Aaron van den Oord and Sander Dieleman and Heiga Zen and Karen Simonyan and Oriol Vinyals and Alexander Graves and Nal Kalchbrenner and Andrew Senior and Koray Kavukcuoglu}, +year = {2016}, +URL = {https://arxiv.org/abs/1609.03499}, +booktitle = {Arxiv} +} + + + + +%%% 1. AE +@inproceedings{Baldi2012, + issn = {0899-7667}, + booktitle = {Proceedings oast the International Conference on Machine Learning (ICML)}, + pages = {37--50}, + title = {{Autoencoders, Unsupervised Learning, and Deep Architectures}}, + author={Pierre Baldi}, + year = {2012} +} + +@inproceedings{vincent2008denoisingae, + title={Extracting and composing robust features with denoising autoencoders}, + author={Vincent, Pascal and Larochelle, Hugo and Bengio, Yoshua and Manzagol, Pierre Antoine}, + booktitle={Proceedings of the international conference on Machine learning (ICML)}, + pages={1096--1103}, + year={2008}, + organization={ACM} +} +@article{vincent2010stacked, + title={Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion}, + author={Vincent, Pascal and Larochelle, Hugo and Lajoie, Isabelle and Bengio, Yoshua and Manzagol, Pierre-Antoine}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={11}, + number={Dec}, + pages={3371--3408}, + year={2010} +} +%%% 2. VAE +@inproceedings{kingma2014auto, + title={Auto-encoding variational bayes}, + author={Kingma, Diederik P and Welling, Max}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2014} +} + + +%%% 3. GANs : DCGAN, ACGAN, Semi-supervised GAN, text2image, InfoGAN, stackGAN, SRGAN +% GAN +@inproceedings{goodfellow2014gan, + title={{Generative Adversarial Nets}}, + author={Goodfellow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2014} +} +% WGAN +@inproceedings{arjovsky2017wasserstein, + title={Wasserstein generative adversarial networks}, + author={Arjovsky, Martin and Chintala, Soumith and Bottou, L{\'e}on}, + booktitle={Processing of the International Conference on Machine Learning (ICML)}, + pages={214--223}, + year={2017} +} +% cGAN +@article{mirza2014conditional, + title={{Conditional Generative Adversarial Nets}}, + author={Mirza, Mehdi and Osindero, Simon}, + journal={arXiv preprint arXiv:1411.1784}, + year={2014} +} + +% ACGAN +@inproceedings{odena2016acgan, + title={Conditional Image Synthesis With Auxiliary Classifier GANs}, + author={Odena, Augustus and Olah, Christopher and Shlens, Jonathon}, + booktitle={Processing of the International Conference on Machine Learning (ICML)}, + year={2017} +} + +% infoGAN +@inproceedings{van2016conditional, + title={{Conditional Image Generation with PixelCNN Decoders}}, + author={Oord, Aaron Van den and Kalchbrenner, Nal and Vinyals, Oriol and Espeholt, Lasse and Graves, Alex and Kavukcuoglu, Koray}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} + +% CycleGAN +@inproceedings{zhu2017cyclegan, + author = {Zhu, Jun-Yan and Park, Taesung and Isola, Phillip and Efros, Alexei A.}, + title = {{Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks}}, + booktitle={Proceedings of International Conference on Computer Vision (ICCV)}, + year = {2017} +} +% DiscoGAN == CycleGAN +@inproceedings{kim2017discogan, + title={Learning to discover cross-domain relations with generative adversarial networks}, + author={Kim, Taeksoo and Cha, Moonsu and Kim, Hyunsoo and Lee, Jungkwon and Kim, Jiwon}, + booktitle={Processing of the International Conference on Machine Learning (ICML)}, + year={2017} +} +％ DualGAN == CycleGAN +@inproceedings{yi2017dualgan, + title={Dualgan: Unsupervised dual learning for image-to-image translation}, + author={Yi, Zili and Zhang, Hao and Tan, Ping and Gong, Minglun}, + booktitle={Proceedings of the International Conference on Computer Vision (ICCV)}, + year={2017} +} +% unsup im2im Advances in Neural Information Processing Systems +@inproceedings{liu2017unsupervised, + title={Unsupervised image-to-image translation networks}, + author={Liu, Ming-Yu and Breuel, Thomas and Kautz, Jan}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={700--708}, + year={2017} +} + +@inproceedings{zhu2017toward, + author = {Zhu, Jun-Yan and Zhang, Richard and Pathak, Deepak and Darrell, Trevor and Efros, Alexei A. and Wang, Oliver and Shechtman, Eli}, + title = {{Toward Multimodal Image-to-Image Translation}}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year = {2017} +} + +% LSGAN +@inproceedings{mao2017lsgan, + title={Least squares generative adversarial networks}, + author={Mao, Xudong and Li, Qing and Xie, Haoran and Lau, Raymond YK and Wang, Zhen and Smolley, Stephen Paul}, + booktitle={Proceedings of the International Conference on Computer Vision (ICCV)}, + pages={2813--2821}, + year={2017}, + organization={IEEE} +} + +% SeqGAN +@inproceedings{yu2017seqgan, + author = {Yu, Lantao and Zhang, Weinan and Wang, Jun and Yu, Yong}, + booktitle = {Association for the Advancement of Artificial Intelligence Conference (AAAI)}, + title = {{SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient}}, + url = {http://arxiv.org/abs/1609.05473}, + year = {2017} +} +% PixelRNN +@inproceedings{oord2016pixel, + title={{Pixel Recurrent Neural Networks}}, + author={Oord, Aaron Van den and Kalchbrenner, Nal and Kavukcuoglu, Koray}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2016} +} +% InfoGAN +@inproceedings{chen2016infogan, + title={{InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets}}, + author={Chen, Xi and Duan, Yan and Houthooft, Rein and Schulman, John and Sutskever, Ilya and Abbeel, Pieter}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} +% GAN speech enhancement +@inproceedings{donahue2018exploring, + title={Exploring speech enhancement with generative adversarial networks for robust speech recognition}, + author={Donahue, Chris and Li, Bo and Prabhavalkar, Rohit}, + booktitle={2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)}, + pages={5024--5028}, + year={2018}, + organization={IEEE} +} + +@inproceedings{odena2016semi, + title={Semi-Supervised Learning with Generative Adversarial Networks}, + author={Odena, Augustus}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML) Workshop}, + year={2016} +} + +% DCGAN +@inproceedings{radford2015dcgan, + title={{Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks}}, + author={Radford, Alec and Metz, Luke and Chintala, Soumith}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2016} +} + +@inproceedings{denton2015deep, + title = {{Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks}}, + author = {Denton, Emily and Chintala, Soumith and Szlam, Arthur and Fergus, Rob}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2015} +} + +% inception score +@inproceedings{salimans2014improved, + title={{Improved Techniques for Training GANs}}, + author={Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} +@inproceedings{heusel2017gans, + title={Gans trained by a two time-scale update rule converge to a local nash equilibrium}, + author={Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2017} +} + +@article{borji2019pros, + title={Pros and cons of GAN evaluation measures}, + author={Borji, Ali}, + journal={Computer Vision and Image Understanding}, + volume={179}, + pages={41--65}, + year={2019}, + publisher={Elsevier} +} +@inproceedings{heusel2017gans, + title={Gans trained by a two time-scale update rule converge to a local nash equilibrium}, + author={Heusel, Martin and Ramsauer, Hubert and Unterthiner, Thomas and Nessler, Bernhard and Hochreiter, Sepp}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={6626--6637}, + year={2017} +} + +% GAN-CLS +@inproceedings{reed2016generative, + title={{Generative Adversarial Text to Image Synthesis}}, + author={Reed, Scott and Akata, Zeynep and Yan, Xinchen and Logeswaran, Lajanugen and Schiele, Bernt and Lee, Honglak}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2016} +} + +@inproceedings{reed2016learning, + title={{Learning What and Where to Draw}}, + author={Reed, Scott and Akata, Zeynep and Mohan, Santosh and Tenka, Samuel and Schiele, Bernt and Lee, Honglak}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} +% StackGAN +@inproceedings{zhang2017stackgan, + title={{StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks}}, + author={Zhang, Han and Xu, Tao and Li, Hongsheng and Zhang, Shaoting and Huang, Xiaolei and Wang, Xiaogang and Metaxas, Dimitris}, + booktitle={Proceedings of the International Conference on Computer Vision (ICCV)}, + year={2017} +} +@article{zhang2017stackgan2, + arxivId = {1710.10916}, + author = {Zhang, Han and Xu, Tao and Li, Hongsheng and Zhang, Shaoting and Wang, Xiaogang and Huang, Xiaolei and Metaxas, Dimitris}, + title = {{StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networks}}, + url = {http://arxiv.org/abs/1710.10916}, + year = {2017} +} + +% SRGAN +@inproceedings{Ledig2017photo, + title={{Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network}}, + author={Ledig, Christian and Theis, Lucas and Huszar, Ferenc and Caballero, Jose and Cunningham, Andrew and Acosta, Alejandro and Aitken, Andrew and Tejani, Alykhan and Totz, Johannes and Wang, Zehan and Shi, Wenzhe}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2017} +} +% SSIM +@article{wang2004image, + title={Image quality assessment: from error visibility to structural similarity}, + author={Wang, Zhou and Bovik, Alan C and Sheikh, Hamid R and Simoncelli, Eero P and others}, + journal={IEEE Transactions on Image Processing (TIP)}, + volume={13}, + number={4}, + pages={600--612}, + year={2004} +} +% Progressive GAN +@inproceedings{karras2017progressive, + title={Progressive growing of GANs for improved quality, stability, and variation}, + author={Karras, Tero and Aila, Timo and Laine, Samuli and Lehtinen, Jaakko}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2018} +} +@inproceedings{collier2018progressively, + title={Progressively Growing Generative Adversarial Networks for High Resolution Semantic Segmentation of Satellite Images}, + author={Collier, Edward and Duffy, Kate and Ganguly, Sangram and Madanguit, Geri and Kalia, Subodh and Shreekant, Gayaka and Nemani, Ramakrishna and Michaelis, Andrew and Li, Shaung and Ganguly, Auroop and others}, + booktitle={Proceedings of the International Conference on Data Mining Workshops (ICDMW)}, + pages={763--769}, + year={2018}, + organization={IEEE} +} + +@inproceedings{Yaniv2017unsupervised, + title={{Unsupervised Cross-Domain Image Generation}}, + author={Taigman, Yaniv and Polyak, Adam and Wolf, Lior}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2017} +} + +@inproceedings{zhu2016generative, + title={{Generative Visual Manipulation on the Natural Image Manifold}}, + author={Zhu, Jun-Yan and Kr{\"a}henb{\"u}hl, Philipp and Shechtman, Eli and Efros, Alexei A.}, + booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, + year={2016} +} + +@inproceedings{li2016Precomputed, + title={{Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks}}, + author={Li, Chuan and Wand, Michael}, + booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, + year={2016} +} + +@inproceedings{Yoo2016pixel, + title={{Pixel-Level Domain Transfer}}, + author={Yoo, Donggeun and Kim, Namil and Park, Sunggyun and Paek, Anthony S. and Kweon, In So}, + booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, + year={2016} +} + +@inproceedings{Brock2017neural, + title={{Neural Photo Editing with Introspective Adversarial Networks}}, + author={Brock, Andrew and Lim, Theodore and Ritchie, J. M. and Weston, Nick}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2017} +} + +@inproceedings{wang2016generative, + title={{Generative Image Modeling Using Style and Structure Adversarial Networks}}, + author={Wang, Xiaolong and Gupta, Abhinav}, + booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, + year={2016} +} + +@inproceedings{perarnau2016icgan, + title={{Invertible Conditional GANs for Image Editing}}, + author={Perarnau, Guim and Weijer, Joost Van De and Raducanu, Bogdan and {\'{A}}lvarez, Jose M and Csiro, Data}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop}, + year={2016} +} + +% BiGAN +@inproceedings{dumoulin2016adversarially, + title={{Adversarially Learned Inference}}, + author={Dumoulin, Vincent and Belghazi, Ishmael and Poole, Ben and Lamb, Alex and Arjovsky, Martin and Mastropietro, Olivier and Courville, Aaron}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2017} +} + +@inproceedings{donahue2016adversarial, + title={{Adversarial Feature Learning}}, + author={Donahue, Jeff and Kr{\"a}henb{\"u}hl, Philipp and Darrell, Trevor}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2017} +} + +% AAE +@inproceedings{makhzani2016aae, + title={{Adversarial Autoencoders}}, + author={Makhzani, Alireza and Shlens, Jonathon and Jaitly, Navdeep and Goodfellow, Ian and Frey, Brendan}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2016} +} + +% VAE+GAN VAE/GAN +@inproceedings{larsen2016vaegan, + title={{Autoencoding Beyond Pixels Using a Learned Similarity Metric}}, + author={Larsen, Anders Boesen Lindbo and S{\o}nderby, S{\o}ren Kaae and Larochelle, Hugo and Winther, Ole}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2016} +} +% GAN collapse ICCV talk -- AlphaGAN +@misc{rosca2017talk, + title = {Solving mode collapse with Autoencoder GAN}, + author = {Rosca, Mihaela and Lakshminarayanan, Balaji and Warde-Farley, David and Mohamed, Shakir}, + url = {{http://elarosca.net/slides/iccv\_autoencoder\_gans.pdf}}, +} +@article{rosca2017alphagan, + title={Variational approaches for auto-encoding generative adversarial networks}, + author={Rosca, Mihaela and Lakshminarayanan, Balaji and Warde-Farley, David and Mohamed, Shakir}, + journal={arXiv preprint arXiv:1706.04987}, + year={2017} +} +% noisy label GAN +@misc{ganhacks, + title = {How to Train a GAN? Tips and tricks to make GANs work}, + author = {Chintala, Soumith and Denton, Emily and Arjovsky, Martin and Mathieu, Michael}, + url = {{https://github.com/soumith/ganhacks}}, +} +@inproceedings{Salimans2016, + author = {Salimans, Tim and Goodfellow, Ian and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + title = {{Improved Techniques for Training GANs}}, + year = {2016} +} + +% unrolled GAN +@inproceedings{metz2016unrolled, + title={Unrolled generative adversarial networks}, + author={Metz, Luke and Poole, Ben and Pfau, David and Sohl-Dickstein, Jascha}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2017} +} +%@misc{kvfrans2016, +% title = {Generative Adversarial Networks Explained}, +% author = {Frans, Kevin}, +% url = {{http://kvfrans.com/generative-adversial-networks-explained/}}, +%} + + +% AGE +@inproceedings{ulyanov2018age, + title={It Takes (Only) Two: Adversarial Generator-Encoder Networks}, + author={Ulyanov, Dmitry and Vedaldi, Andrea and Lempitsky, Victor}, + booktitle={Association for the Advancement of Artificial Intelligence Conference (AAAI)}, + year={2018} +} +% other VAE/GAN +@article{mescheder2017adversarial, + title={Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks}, + author={Mescheder, Lars and Nowozin, Sebastian and Geiger, Andreas}, + journal={arXiv preprint arXiv:1701.04722}, + year={2017} +} + +% residual net resnet +@inproceedings{he2016deep, + title={{Deep Residual Learning for Image Recognition}}, + author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2016} +} + +@inproceedings{gurumurthy2017deligan, + title={{DeliGAN: Generative adversarial networks for diverse and limited data}}, + author={Gurumurthy, Swaminathan and Sarvadevabhatla, Ravi Kiran and Radhakrishnan, V Babu}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + volume={1}, + year={2017} +} + +% Style Transfer and Super Resolution +@inproceedings{johnson2016perceptual, + title={{Perceptual Losses for Real-Time Style Transfer and Super-Resolution}}, + author={Johnson, Justin and Alahi, Alexandre and Fei-Fei, Li}, + booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, + year={2016}, +} + +@book{Nasrollahi2014, + author = {Nasrollahi, Kamal and Moeslund, Thomas B.}, + booktitle = {Machine Vision and Applications}, + doi = {10.1007/s00138-014-0623-4}, + eprint = {1610.04490}, + pages = {1423--1468}, + title = {{Super-resolution: A comprehensive survey}}, + volume = {25}, + year = {2014} +} + +@article{Zou2012, + author = {Zou, Wilman}, + journal = {IEEE Transactions on Image Processing (TIP}, + number = {3}, + pages = {4408--4410}, + title = {{Very Low Resolution Face Recognition in Parallel Environment}}, + volume = {3}, + year = {2012} +} + +@inproceedings{Yang2007, + author = {Yang, Qingxiong and Yang, Ruigang and Davis, James and Nist�r, David}, + doi = {10.1109/CVPR.2007.383211}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + title = {{Spatial-Depth Supre Resolution for Range Images}}, + year = {2007} +} + +@inproceedings{Lotter2016, + author = {Lotter, William and Kreiman, Gabriel and Cox, David}, + booktitle = {Proceedings of the International Conference on Learning Representations (ICLR)}, + number = {2015}, + pages = {1--12}, + title = {{Unsupervised Learning of Visual Structure using Predictive Generative Networks}}, + year = {2016} +} + +%% 4. Other Generative Papers +@article{oord2016pixelcnn, + title={Conditional image generation with PixelCNN decoders}, + author={Oord, Aaron van den and Kalchbrenner, Nal and Vinyals, Oriol and Espeholt, Lasse and Graves, Alex and Kavukcuoglu, Koray}, + journal={arXiv preprint arXiv:1606.05328}, + year={2016} +} + +@inproceedings{nguyen2016plug, + title={Plug \& Play Generative Networks: Conditional Iterative Generation of Images in Latent Space}, + author={Nguyen, Anh and Yosinski, Jason and Bengio, Yoshua and Dosovitskiy, Alexey and Clune, Jeff}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2017} +} + +@inproceedings{van2016conditional, + title={{Conditional Image Generation with PixelCNN Decoders}}, + author={Oord, Aaron Van den and Kalchbrenner, Nal and Vinyals, Oriol and Espeholt, Lasse and Graves, Alex and Kavukcuoglu, Koray}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} + +@inproceedings{oord2016pixel, + title={{Pixel Recurrent Neural Networks}}, + author={Oord, Aaron Van den and Kalchbrenner, Nal and Kavukcuoglu, Koray}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2016} +} + +@inproceedings{reed2016controlgenerate, + title={GENERATING INTERPRETABLE IMAGES WITH CONTROLLABLE STRUCTURE}, + author={Reed, Scott and van den Oord, A and Kalchbrenner, N and Bapst, V and Botvinick, M and de Freitas, N}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2017} +} + +@inproceedings{reed2016whatwhere, + title={Learning what and where to draw}, + author={Reed, Scott and Akata, Zeynep and Mohan, Santosh and Tenka, Samuel and Schiele, Bernt and Lee, Honglak}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} + +@inproceedings{gregor2014draw, + author = {Gregor, Karol and Danihelka, Ivo and Graves, Alex and Wierstra, Daan}, + title = {{DRAW: A Recurrent Neural Network For Image Generation}}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year = {2015} +} + + +%% 5. Image Captioning +@article{vinyals2016show, + title={Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge}, + author={Vinyals, Oriol and Toshev, Alexander and Bengio, Samy and Erhan, Dumitru}, + year={2016}, + journal={IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)} +} +@inproceedings{xu2015show, + title={Show, attend and tell: Neural image caption generation with visual attention}, + author={Xu, Kelvin and Ba, Jimmy and Kiros, Ryan and Cho, Kyunghyun and Courville, Aaron and Salakhudinov, Ruslan and Zemel, Rich and Bengio, Yoshua}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + pages={2048--2057}, + year={2015} +} +@misc{tensorflowmodel_im2txt, + title = {TensorFlow models : Show and Tell: A Neural Image Caption Generator}, + author = {Chris, Shallue}, + url = {{https://github.com/tensorflow/models/tree/master/research/im2txt}}, +} + +@inproceedings{karpathy2015deep, + title={Deep visual-semantic alignments for generating image descriptions}, + author={Karpathy, Andrej and Fei-Fei, Li}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2015} +} + +@inproceedings{antol2016vqa, + author = {Antol, Stanislaw and Agrawal, Aishwarya and Lu, Jiasen and Mitchell, Margaret and Batra, Dhruv and Zitnick, C. Lawrence and Parikh, Devi}, + doi = {10.1109/ICCV.2015.279}, + booktitle = {Proceedings of the IEEE International Conference on Computer Vision (ICCV)}, + pages = {2425--2433}, + title = {{VQA: Visual question answering}}, + volume = {11-18-Dece}, + year = {2016} +} + +@inproceedings{kulkarni2011baby, + title={Baby talk: Understanding and generating image descriptions}, + author={Kulkarni, Girish and Premraj, Visruth and Dhar, Sagnik and Li, Siming and Choi, Yejin and Berg, Alexander C and Berg, Tamara L}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2011}, + organization={Citeseer} +} +@inproceedings{farhadi2010every, + title={Every picture tells a story: Generating sentences from images}, + author={Farhadi, Ali and Hejrati, Mohsen and Sadeghi, Mohammad Amin and Young, Peter and Rashtchian, Cyrus and Hockenmaier, Julia and Forsyth, David}, + booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, + pages={15--29}, + year={2010}, + organization={Springer} +} +%% 6. Text-image mapping and Word embedding +@article{kiros2015unifying, + title={Unifying visual-semantic embeddings with multimodal neural language models}, + author={Kiros, Ryan and Salakhutdinov, Ruslan and Zemel, Richard S}, + journal={Transactions of the Association for Computational Linguistics (TACL)}, + year={2015} +} + +@inproceedings{karpathy2014txtimmap, + title={Deep fragment embeddings for bidirectional image sentence mapping}, + author={Karpathy, Andrej and Joulin, Armand and Li, Fei Fei F}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2014} +} + +@inproceedings{mikolov2013distributed, + title={Distributed representations of words and phrases and their compositionality}, + author={Mikolov, Tomas and Sutskever, Ilya and Chen, Kai and Corrado, Greg S and Dean, Jeff}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={3111--3119}, + year={2013} +} + +@inproceedings{reed2016learningdeep, + title={{Learning Deep Representations of Fine-Grained Visual Descriptions}}, + author={Reed, Scott and Akata, Zeynep and Lee, Honglak and Schiele, Bernt}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2016} +} + +@article{rong2014word2vec, + title={Word2vec parameter learning explained}, + author={Rong, Xin}, + journal={arXiv preprint arXiv:1411.2738}, + year={2014} +} +% word2vec original paper +@article{mikolov2013efficient, + title={Efficient estimation of word representations in vector space}, + author={Mikolov, Tomas and Chen, Kai and Corrado, Greg and Dean, Jeffrey}, + journal={Computing Research Repository (CoRR)}, + year={2013} +} +@inproceedings{pennington2014glove, + title={Glove: Global vectors for word representation}, + author={Pennington, Jeffrey and Socher, Richard and Manning, Christopher}, + booktitle={Proceedings of the Empirical Methods in Natural Language Processing (EMNLP) Conference}, + pages={1532--1543}, + year={2014} +} +@article{zhang2010understanding, + title={Understanding bag-of-words model: a statistical framework}, + author={Zhang, Yin and Jin, Rong and Zhou, Zhi-Hua}, + journal={International Journal of Machine Learning and Cybernetics}, + volume={1}, + number={1-4}, + pages={43--52}, + year={2010}, + publisher={Springer} +} +%% 7. CNN architecture sub-pixel resize convolution +@inproceedings{szegedy2015inception, + title={Rethinking the inception architecture for computer vision}, + author={Szegedy, Christian and Vanhoucke, Vincent and Ioffe, Sergey and Shlens, Jonathon and Wojna, Zbigniew}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2016} +} + +@article{odena2016resizeconv, + title={Rethinking the inception architecture for computer vision}, + author={Odena, Augustus and Vincent, Dumoulinand Chris, Olah}, + journal={Distill}, + year={2016} +} + + +%% 8. CNN unsupervised for 3D structure +@article{rezende2016unsupervised, + title={Unsupervised learning of 3d structure from images}, + author={Rezende, Danilo Jimenez and Eslami, SM and Mohamed, Shakir and Battaglia, Peter and Jaderberg, Max and Heess, Nicolas}, + journal={arXiv preprint arXiv:1607.00662}, + year={2016} +} + +%% 9. Traditional Im2im +@article{bitouk2008face, + title={Face swapping: automatically replacing faces in photographs}, + author={Bitouk, Dmitri and Kumar, Neeraj and Dhillon, Samreen and Belhumeur, Peter and Nayar, Shree K}, + journal={ACM Transactions on Graphics (TOG)}, + volume={27}, + number={3}, + pages={39}, + year={2008}, + publisher={ACM} +} + +@article{korshunova2016fast, + title={Fast Face-swap Using Convolutional Neural Networks}, + author={Korshunova, Iryna and Shi, Wenzhe and Dambre, Joni and Theis, Lucas}, + journal={arXiv preprint arXiv:1611.09577}, + year={2016} +} + +@article{chen2009sketch2photo, + title={Sketch2Photo: internet image montage}, + author={Chen, Tao and Cheng, Ming-Ming and Tan, Ping and Shamir, Ariel and Hu, Shi-Min}, + journal={ACM Transactions on Graphics (TOG)}, + volume={28}, + number={5}, + pages={124}, + year={2009}, + publisher={ACM} +} + +@article{shih2013data, + title={Data-driven hallucination of different times of day from a single outdoor photo}, + author={Shih, Yichang and Paris, Sylvain and Durand, Fr{\'e}do and Freeman, William T}, + journal={ACM Transactions on Graphics (TOG)}, + volume={32}, + number={6}, + pages={200}, + year={2013}, + publisher={ACM} +} + +@article{laffont2014transient, + title={Transient attributes for high-level understanding and editing of outdoor scenes}, + author={Laffont, Pierre-Yves and Ren, Zhile and Tao, Xiaofeng and Qian, Chao and Hays, James}, + journal={ACM Transactions on Graphics (TOG)}, + volume={33}, + number={4}, + pages={149}, + year={2014}, + publisher={ACM} +} + +@inproceedings{taigman2016unsupervised, + title={Unsupervised Cross-Domain Image Generation}, + author={Taigman, Yaniv and Polyak, Adam and Wolf, Lior}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2017} +} + +@article{Hochreiter1997lstm, + author = {Hochreiter, Sepp and Hochreiter, S and Schmidhuber, J{\"{u}}rgen and Schmidhuber, J}, + isbn = {08997667 (ISSN)}, + issn = {0899-7667}, + journal = {Neural Computation}, + number = {8}, + pages = {1735--80}, + pmid = {9377276}, + title = {{Long Short-Term Memory.}}, + volume = {9}, + year = {1997} +} + +@article{greff2017lstm, + title={{LSTM}: A search space odyssey}, + author={Greff, Klaus and Srivastava, Rupesh K and Koutn{\'\i}k, Jan and Steunebrink, Bas R and Schmidhuber, J{\"u}rgen}, + journal={IEEE Transactions on Neural Networks and Learning Systems (TNNLS)}, + volume={28}, + number={10}, + pages={2222--2232}, + year={2017}, + publisher={IEEE} +} + +%%% +@inproceedings{girshick2015fastrcnn, + title={{Fast R-CNN}}, + author={Girshick, Ross}, + booktitle={Proceedings of the IEEE International Conference on Computer Vision (ICCV)}, + pages={1440--1448}, + year={2015} +} + +%% domain adaptation +@inproceedings{zhao2018adversarial, + title={Adversarial multiple source domain adaptation}, + author={Zhao, Han and Zhang, Shanghang and Wu, Guanhang and Moura, Jos{\'e} MF and Costeira, Joao P and Gordon, Geoffrey J}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={8568--8579}, + year={2018} +} + +%%% Seq2Seq language translation GRU +@inproceedings{cho2014learning, + title={Learning phrase representations using {RNN} encoder-decoder for statistical machine translation}, + author={Cho, Kyunghyun and Van Merri{\"e}nboer, Bart and Gulcehre, Caglar and Bahdanau, Dzmitry and Bougares, Fethi and Schwenk, Holger and Bengio, Yoshua}, + booktitle={Proceedings of the Empirical Methods in Natural Language Processing (EMNLP) Conference}, + year={2014} +} + +@inproceedings{jozefowicz2015empirical, + title={An empirical exploration of recurrent network architectures}, + author={Jozefowicz, Rafal and Zaremba, Wojciech and Sutskever, Ilya}, + booktitle={International Conference on Machine Learning}, + pages={2342--2350}, + year={2015} +} + +@inproceedings{sutskever2014sequence, + title={Sequence to sequence learning with neural networks}, + author={Sutskever, Ilya and Vinyals, Oriol and Le, Quoc V}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={3104--3112}, + year={2014} +} + +@inproceedings{rezende2014stochastic, + title={{Stochastic Backpropagation and Approximate Inference in Deep Generative Models}}, + author={Rezende, Danilo Jimenez and Mohamed, Shakir and Wierstra, Daan}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2014} +} + +@inproceedings{liao2018deep, + title={Deep sequence learning with auxiliary information for traffic prediction}, + author={Liao, Binbing and Zhang, Jingqing and Wu, Chao and McIlwraith, Douglas and Chen, Tong and Yang, Shengwen and Guo, Yike and Wu, Fei}, + booktitle={Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery \& Data Mining}, + pages={537--546}, + year={2018}, + organization={ACM} +} + +@inproceedings{liao2018dest, + title={{Dest-ResNet}: A Deep Spatiotemporal Residual Network for Hotspot Traffic Speed Prediction}, + author={Liao, Binbing and Zhang, Jingqing and Cai, Ming and Tang, Siliang and Gao, Yifan and Wu, Chao and Yang, Shengwen and Zhu, Wenwu and Guo, Yike and Wu, Fei}, + booktitle={2018 ACM Multimedia Conference on Multimedia Conference}, + pages={1883--1891}, + year={2018}, + organization={ACM} +} + +@article{song2019arena, + title={Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence}, + author={Song, Yuhang and Wang, Jianyi and Lukasiewicz, Thomas and Xu, Zhenghua and Xu, Mai and Ding, Zihan and Wu, Lianlong}, + journal={arXiv preprint arXiv:1905.08085}, + year={2019} +} + + +@inproceedings{ +peterliu2018generating, +title={Generating Wikipedia by Summarizing Long Sequences}, +author={Peter J. Liu and Mohammad Saleh and Etienne Pot and Ben Goodrich and Ryan Sepassi and Lukasz Kaiser and Noam Shazeer}, +booktitle={International Conference on Learning Representations}, +year={2018}, +url={https://openreview.net/forum?id=Hyg0vbWC-}, +} + +@article{zhang2019pegasus, + title={{PEGASUS}: Pre-training with Extracted Gap-sentences for Abstractive Summarization}, + author={Zhang, Jingqing and Zhao, Yao and Saleh, Mohammad and Liu, Peter J}, + journal={arXiv preprint arXiv:1912.08777}, + year={2019} +} + + +%%% generative model vs discriminative model +@inproceedings{ng2002discriminative, + title={On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes}, + author={Ng, Andrew Y and Jordan, Michael I}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={841--848}, + year={2002} +} + +@inproceedings{rish2001empirical, + title={An empirical study of the naive Bayes classifier}, + author={Rish, Irina and others}, + booktitle={International Joint Conference on Artificial Intelligence 2001 workshop on empirical methods in artificial intelligence}, + volume={3}, + number={22}, + pages={41--46}, + year={2001} +} + +%%% Computer Vision Tasks CV +@inproceedings{redmon2017yolo9000, + title={YOLO9000: better, faster, stronger}, + author={Redmon, Joseph and Farhadi, Ali}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2017} +} + +@inproceedings{long2015fully, + title={Fully convolutional networks for semantic segmentation}, + author={Long, Jonathan and Shelhamer, Evan and Darrell, Trevor}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + pages={3431--3440}, + year={2015} +} + +@inproceedings{noh2015learning, + title={Learning deconvolution network for semantic segmentation}, + author={Noh, Hyeonwoo and Hong, Seunghoon and Han, Bohyung}, + booktitle={Proceedings of the International Conference on Computer Vision (ICCV)}, + pages={1520--1528}, + year={2015} +} + +@inproceedings{cao2017realtime, + author = {Cao, Zhe and Simon, Zhe and Wei, Shih-En and Sheikh, Shih-En}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields}, + year = {2017} +} + +@inproceedings{he2017mask, + title={Mask R-CNN}, + author={He, Kaiming and Gkioxari, Georgia and Doll{\'a}r, Piotr and Girshick, Ross}, + booktitle={Proceedings of the International Conference on Computer Vision (ICCV)}, + pages={2980--2988}, + year={2017}, + organization={IEEE} +} + +@inproceedings{korshunova2017fast, + title={Fast face-swap using convolutional neural networks}, + author={Korshunova, Iryna and Shi, Wenzhe and Dambre, Joni and Theis, Lucas}, + booktitle={Proceedings of the International Conference on Computer Vision (ICCV)}, + year={2017} +} + +@inproceedings{xie2012image, + title={Image denoising and inpainting with deep neural networks}, + author={Xie, Junyuan and Xu, Linli and Chen, Enhong}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems}, + pages={341--349}, + year={2012} +} + +@inproceedings{liu2015deep, + title={Deep convolutional neural fields for depth estimation from a single image}, + author={Liu, Fayao and Shen, Chunhua and Lin, Guosheng}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + pages={5162--5170}, + year={2015} +} + +@article{shih2013data, + title={Data-driven hallucination of different times of day from a single outdoor photo}, + author={Shih, Yichang and Paris, Sylvain and Durand, Fr{\'e}do and Freeman, William T}, + journal={ACM Transactions on Graphics (TOG)}, + volume={32}, + number={6}, + pages={200}, + year={2013}, + publisher={ACM} +} + +@article{laffont2014transient, + title={Transient attributes for high-level understanding and editing of outdoor scenes}, + author={Laffont, Pierre-Yves and Ren, Zhile and Tao, Xiaofeng and Qian, Chao and Hays, James}, + journal={ACM Transactions on Graphics (TOG)}, + volume={33}, + number={4}, + pages={149}, + year={2014}, + publisher={ACM} +} + +@article{chen2009sketch2photo, + title={Sketch2photo: Internet image montage}, + author={Chen, Tao and Cheng, Ming-Ming and Tan, Ping and Shamir, Ariel and Hu, Shi-Min}, + journal={ACM Transactions on Graphics (TOG)}, + volume={28}, + number={5}, + pages={124}, + year={2009}, + organization={ACM} +} + +%%% understand CNN + +@article{mahendran2016visualizing, + title={Visualizing deep convolutional neural networks using natural pre-images}, + author={Mahendran, Aravindh and Vedaldi, Andrea}, + journal={International Journal of Computer Vision (IJCV)}, + volume={120}, + number={3}, + pages={233--255}, + year={2016}, + publisher={Springer} +} + +@article{dumoulin2016guide, + title={A guide to convolution arithmetic for deep learning}, + author={Dumoulin, Vincent and Visin, Francesco}, + journal={arXiv preprint arXiv:1603.07285}, + year={2016} +} + +@article{yin2017comparative, + title={Comparative study of CNN and RNN for natural language processing}, + author={Yin, Wenpeng and Kann, Katharina and Yu, Mo and Sch{\"u}tze, Hinrich}, + journal={arXiv preprint arXiv:1702.01923}, + year={2017} +} + + + +%%% understand rnn + +@article{chung2014empirical, + title={Empirical evaluation of gated recurrent neural networks on sequence modeling}, + author={Chung, Junyoung and Gulcehre, Caglar and Cho, KyungHyun and Bengio, Yoshua}, + journal={arXiv preprint arXiv:1412.3555}, + year={2014} +} + +@inproceedings{luong-etal-2015-effective, + title = "Effective Approaches to Attention-based Neural Machine Translation", + author = "Luong, Thang and + Pham, Hieu and + Manning, Christopher D.", + booktitle = "Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing", + month = sep, + year = "2015", + address = "Lisbon, Portugal", + publisher = "Association for Computational Linguistics", + url = "https://www.aclweb.org/anthology/D15-1166", + doi = "10.18653/v1/D15-1166", + pages = "1412--1421", +} + +@inproceedings{nallapati2017summarunner, + author = {Nallapati, Ramesh and Zhai, Feifei and Zhou, Bowen}, + title = {SummaRuNNer: A Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents}, + year = {2017}, + publisher = {AAAI Press}, + booktitle = {Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence}, + pages = {3075–3081}, + numpages = {7}, + location = {San Francisco, California, USA}, + series = {AAAI’17} +} + +@inproceedings{zhang-etal-2019-integrating, + title = "Integrating Semantic Knowledge to Tackle Zero-shot Text Classification", + author = "Zhang, Jingqing and + Lertvittayakumjorn, Piyawat and + Guo, Yike", + booktitle = "Proceedings of the 2019 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)", + month = jun, + year = "2019", + address = "Minneapolis, Minnesota", + publisher = "Association for Computational Linguistics", + url = "https://www.aclweb.org/anthology/N19-1108", + doi = "10.18653/v1/N19-1108", + pages = "1031--1040", + abstract = "Insufficient or even unavailable training data of emerging classes is a big challenge of many classification tasks, including text classification. Recognising text documents of classes that have never been seen in the learning stage, so-called zero-shot text classification, is therefore difficult and only limited previous works tackled this problem. In this paper, we propose a two-phase framework together with data augmentation and feature augmentation to solve this problem. Four kinds of semantic knowledge (word embeddings, class descriptions, class hierarchy, and a general knowledge graph) are incorporated into the proposed framework to deal with instances of unseen classes effectively. Experimental results show that each and the combination of the two phases achieve the best overall accuracy compared with baselines and recent approaches in classifying real-world texts under the zero-shot scenario.", +} + +@inproceedings{lee-dernoncourt-2016-sequential, + title = "Sequential Short-Text Classification with Recurrent and Convolutional Neural Networks", + author = "Lee, Ji Young and + Dernoncourt, Franck", + booktitle = "Proceedings of the 2016 Conference of the North {A}merican Chapter of the Association for Computational Linguistics: Human Language Technologies", + month = jun, + year = "2016", + address = "San Diego, California", + publisher = "Association for Computational Linguistics", + url = "https://www.aclweb.org/anthology/N16-1062", + doi = "10.18653/v1/N16-1062", + pages = "515--520", +} + +@article{wierstra2010recurrent, + title={Recurrent policy gradients}, + author={Wierstra, Daan and F{\"o}rster, Alexander and Peters, Jan and Schmidhuber, J{\"u}rgen}, + journal={Logic Journal of the IGPL}, + volume={18}, + number={5}, + pages={620--634}, + year={2010}, + publisher={Oxford University Press} +} + + +%%% Transfer learning +@inproceedings{pratt1993discriminability, + title={Discriminability-based transfer between neural networks}, + author={Pratt, Lorien Y}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={204--211}, + year={1993} +} +@article{west2007spring, + title={Spring research presentation: A theoretical foundation for inductive transfer}, + author={West, Jeremy and Ventura, Dan and Warnick, Sean}, + journal={Brigham Young University, College of Physical and Mathematical Sciences}, + volume={1}, + year={2007} +} + +%%% RL reinforcement alphago +@article{silver2017master, + title={Mastering the game of Go without human knowledge}, + author={Silver, David and Schrittwieser, Julian and Simonyan, Karen and Antonoglou, Ioannis and Huang, Aja and Guez, Arthur and Hubert, Thomas and Baker, Lucas and Lai, Matthew and Bolton, Adrian and Yutian Chen and Timothy Lillicrap and Fan Hui and Laurent Sifre and George van den Driessche and Thore Graepel and Hassabis, Demis}, + journal={Nature}, + volume={550}, + number={7676}, + pages={354}, + year={2017}, + publisher={Nature Publishing Group} +} + +@misc{pingpixel2016, + author={Andrej Karpathy}, + title = {Deep Reinforcement Learning: Pong from Pixels}, + year = {2016}, + howpublished={\url{http://karpathy.github.io/2016/05/31/rl/}}, +} + +@article{mnih2015human, + title={Human-level control through deep reinforcement learning}, + author={Volodymyr Mnih and Koray Kavukcuoglu and David Silver and Andrei A. Rusu and Joel Veness and Marc G. Bellemare and Alex Graves and Martin Riedmiller and Andreas K. Fidjeland and Georg Ostrovski and Stig Petersen and Charles Beattie and Amir Sadik and Ioannis Antonoglou and Helen King and Dharshan Kumaran and Daan Wierstra and Shane Legg and Demis Hassabis}, + journal={Nature}, + year={2015} +} + +@article{williams1992REINFORCE, + title={Simple statistical gradient-following algorithms for connectionist reinforcement learning}, + author={Williams, Ronald J}, + journal={Machine Learning}, + volume={8}, + number={3-4}, + pages={229--256}, + year={1992}, + publisher={Springer} +} + +%%% Book +@book{goodfellowDLbook2016, + title={Deep Learning}, + author={Ian Goodfellow and Yoshua Bengio and Aaron Courville}, + publisher={MIT Press}, + note={\url{http://www.deeplearningbook.org}}, + year={2016} +} +@book{affine, + title={Affine transformation}, + author={Hazewinkel, Michiel, ed.}, + publisher={Springer}, + year={2001} +} +@book{meyer1992wavelets, + title={Wavelets and applications}, + author={Meyer, Yves}, + volume={31}, + year={1992}, + publisher={Masson Paris} +} + +%% +@article{maaten2008tsne, + title={Visualizing data using t-SNE}, + author={Maaten, Laurens van der and Hinton, Geoffrey}, + journal={Journal of Machine Learning Research}, + volume={9}, + number={Nov}, + pages={2579--2605}, + year={2008} +} +%% image inpainting +@inproceedings{yeh2017inpainting, + title={Semantic image inpainting with deep generative models}, + author={Yeh, Raymond A and Chen, Chen and Yian Lim, Teck and Schwing, Alexander G and Hasegawa-Johnson, Mark and Do, Minh N}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + pages={5485--5493}, + year={2017} +} +@inproceedings{pathak2016context, + title={Context encoders: Feature learning by inpainting}, + author={Pathak, Deepak and Krahenbuhl, Philipp and Donahue, Jeff and Darrell, Trevor and Efros, Alexei A}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + pages={2536--2544}, + year={2016} +} +@inproceedings{barnes2009patchmatch, + title={PatchMatch: A randomized correspondence algorithm for structural image editing}, + author={Barnes, Connelly and Shechtman, Eli and Finkelstein, Adam and Goldman, Dan B}, + booktitle={ACM Transactions on Graphics (ToG)}, + volume={28}, + number={3}, + pages={24}, + year={2009}, + organization={ACM} +} +@article{hu2013lowrank, + title={Fast and accurate matrix completion via truncated nuclear norm regularization}, + author={Hu, Yao and Zhang, Debing and Ye, Jieping and Li, Xuelong and He, Xiaofei}, + journal={IEEE transactions on pattern analysis and machine intelligence}, + volume={35}, + number={9}, + pages={2117--2130}, + year={2013}, + publisher={IEEE} +} +%%% Datasets : MSCOCO, Imagenet, SVHN, CUB-bird, flower +@inproceedings{lin2014microsoft, + title={Microsoft COCO: Common objects in context}, + author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence}, + journal={Proceedings of the European Conference on Computer Vision (ECCV)}, + year={2014}, + organization={Springer} +} + +@inproceedings{liu2015faceattributes, + author = {Ziwei Liu and Ping Luo and Xiaogang Wang and Xiaoou Tang}, + title = {Deep Learning Face Attributes in the Wild}, + booktitle = {Proceedings of the International Conference on Computer Vision (ICCV)}, + month = December, + year = {2015} +} + +@inproceedings{netzer2011svhn, + title={Reading digits in natural images with unsupervised feature learning}, + author={Netzer, Yuval and Wang, Tao and Coates, Adam and Bissacco, Alessandro and Wu, Bo and Ng, Andrew Y}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop on Deep Learning and Unsupervised Feature Learning}, + volume={2011}, + number={2}, + pages={5}, + year={2011} +} + +@techreport{WahCUB_200_2011, + Title = {{The Caltech-UCSD Birds-200-2011 Dataset}}, + Author = {Wah, C. and Branson, S. and Welinder, P. and Perona, P. and Belongie, S.}, + Year = {2011}, + Institution = {California Institute of Technology}, + Number = {CNS-TR-2011-001} +} + +@inproceedings{nilsback2008flower, + title={{Automated Flower Classification over a Large Number of Classes}}, + author={Nilsback, Maria-Elena and Zisserman, Andrew}, + booktitle={Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing (ICCVGIP)}, + year={2008} +} + +@article{bellman1954theory, + title={The theory of dynamic programming}, + author={Bellman, Richard and others}, + journal={Bulletin of the American Mathematical Society}, + volume={60}, + number={6}, + pages={503--515}, + year={1954}, + publisher={American Mathematical Society} +} + +@article{russakovsky2015imagenet, + title={{Imagenet Large Scale Visual Recognition Challenge}}, + author={Russakovsky, Olga and Deng, Jia and Su, Hao and Krause, Jonathan and Satheesh, Sanjeev and Ma, Sean and Huang, Zhiheng and Karpathy, Andrej and Khosla, Aditya and Bernstein, Michael}, + journal={International Journal of Computer Vision (IJCV)}, + volume={115}, + number={3}, + pages={211--252}, + year={2015}, + publisher={Springer} +} + +@article{krishna2017visual, + title={Visual genome: Connecting language and vision using crowdsourced dense image annotations}, + author={Krishna, Ranjay and Zhu, Yuke and Groth, Oliver and Johnson, Justin and Hata, Kenji and Kravitz, Joshua and Chen, Stephanie and Kalantidis, Yannis and Li, Li-Jia and Shamma, David A and others}, + journal={International Journal of Computer Vision}, + volume={123}, + number={1}, + pages={32--73}, + year={2017}, + publisher={Springer} +} + +%%% Dataset : IXI MICCAI + +@misc{miccaidataset, + title = {MICCAI 2013 grand challenge dataset}, + author = {}, + url = {{http://masiweb.vuse.vanderbilt.edu/workshop2013/index.php/Segmentation\_Challenge\_Details}}, +} + +@misc{ixidataset, + title = {IXI – Information eXtraction from Images}, + author = {Amir, Alansary and Paul, Aljabar and Gareth, Ball and etc}, + url = {{http://brain-development.org/ixi-dataset/}}, +} + +% MPII +@inproceedings{andriluka14cvpr, + title={2D Human Pose Estimation: New Benchmark and State of the Art Analysis}, + author={Mykhaylo Andriluka and Leonid Pishchulin and Peter Gehler and Schiele, Bernt}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2014}, +} + +%% Object detection +@inproceedings{li2017perceptual, + title={Perceptual generative adversarial networks for small object detection}, + author={Li, Jianan and Liang, Xiaodan and Wei, Yunchao and Xu, Tingfa and Feng, Jiashi and Yan, Shuicheng}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + pages={1222--1230}, + year={2017} +} + +%%% Domain Adaptation +@article{ganin2016domain, + title={Domain-adversarial training of neural networks}, + author={Ganin, Yaroslav and Ustinova, Evgeniya and Ajakan, Hana and Germain, Pascal and Larochelle, Hugo and Laviolette, Fran{\c{c}}ois and Marchand, Mario and Lempitsky, Victor}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={17}, + number={59}, + pages={1--35}, + year={2016} +} + +@inproceedings{chen2012marginalized, + title={Marginalized denoising autoencoders for domain adaptation}, + author={Chen, Minmin and Xu, Zhixiang and Weinberger, Kilian and Sha, Fei}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2012} +} +@inproceedings{shanghang2018multiple, + title={Multiple source domain adaptation with adversarial learning}, + author={Zhao, Han and Zhang, Shanghang and Wu, Guanhang and Gordon, Geoffrey J and others}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, year={2018} +} + +@book{hawkins2007intelligence, + title={On intelligence}, + author={Hawkins, Jeff and Blakeslee, Sandra}, + year={2007}, + publisher={Macmillan} +} + +%% +@inproceedings{dosovitskiy2015learning, + title={{Learning to Generate Chairs with Convolutional Neural Networks}}, + author={Dosovitskiy, Alexey and Tobias Springenberg, Jost and Brox, Thomas}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2015} +} + +@inproceedings{yang2015weakly, + title={{Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis}}, + author={Yang, Jimei and Reed, Scott and Yang, Ming-Hsuan and Lee, Honglak}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2015} +} + +@inproceedings{reed2015deepVisual, + title={{Deep Visual Analogy-Making}}, + author={Reed, Scott and Zhang, Yi and Zhang, Yuting and Lee, Honglak}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2015} +} + +@inproceedings{gregor2015draw, + title={Stochastic Backpropagation and Approximate Inference in Deep Generative Models}, + author={Gregor, Karol and Danihelka, Ivo and Graves, Alex and Rezende, Danilo Jimenez and Wierstra, Daan}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2015} +} + +@inproceedings{mansimov2016, + author = {Mansimov, Elman and Parisotto, Emilio and Ba, Jimmy Lei and Salakhutdinov, Ruslan}, + booktitle = {Proceedings of the International Conference on Learning Representations (ICLR)}, + title = {{Generating Images from Captions with Attention}}, + year = {2016} +} + + +@inproceedings{yan2016attribute2image, + title={{Attribute2Image: Conditional Image Generation from Visual Attributes}}, + author={Yan, Xinchen and Yang, Jimei and Sohn, Kihyuk and Lee, Honglak}, + booktitle={Proceedings of the European Conference on Computer Vision (ECCV)}, + year={2016} +} +@inproceedings{niklaus2017video, + title={Video frame interpolation via adaptive separable convolution}, + author={Niklaus, Simon and Mai, Long and Liu, Feng}, + booktitle={Proceedings of the IEEE International Conference on Computer Vision}, + pages={261--270}, + year={2017} +} + +%%% Training, Optimizer, Batch Normalization +@inproceedings{KingmaAdam2014, + title = {{Adam}: A Method for Stochastic Optimization}, + author = {Kingma, Diederik and Ba, Jimmy}, + booktitle = {Proceedings of the International Conference on Learning Representations (ICLR)}, + year = {2014} +} +@article{duchi2011adagrad, + title={Adaptive subgradient methods for online learning and stochastic optimization}, + author={Duchi, John and Hazan, Elad and Singer, Yoram}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={12}, + number={Jul}, + pages={2121--2159}, + year={2011} +} +@techreport{tieleman2012rmsprop, + title={Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning}, + author={Tieleman, T and Hinton, G}, + year={2017}, + institution={Technical Report} +} +@inproceedings{ioffe2015batch, + title={Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift}, + author={Ioffe, Sergey and Szegedy, Christian}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2015}, +} + +%%% Tools: Scikit-learn, Gensim, NLTK +@article{pedregosa2011scikit, + title={{Scikit-learn: Machine learning in Python}}, + author={Pedregosa, Fabian and Varoquaux, Ga{\"e}l and Gramfort, Alexandre and Michel, Vincent and Thirion, Bertrand and Grisel, Olivier and Blondel, Mathieu and Prettenhofer, Peter and Weiss, Ron and Dubourg, Vincent and Jake Vanderplas and Alexandre Passos and David Cournapeau and Matthieu Brucher and Matthieu Perrot and Edouard Duchesnay}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={12}, + number={Oct}, + pages={2825--2830}, + year={2011} +} +@inproceedings{rehurek2010gensim, + title={Software framework for topic modelling with large corpora}, + author={Rehurek, Radim and Sojka, Petr}, + booktitle={In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks}, + year={2010}, + organization={Citeseer} +} +@inproceedings{bird2004nltk, + title={NLTK: the natural language toolkit}, + author={Bird, Steven and Loper, Edward}, + booktitle={Proceedings of the ACL 2004 on Interactive poster and demonstration sessions}, + pages={31}, + year={2004}, + organization={Association for Computational Linguistics} +} +%% Tool: Spark, Hadoop +@article{zaharia2010spark, + title={Spark: Cluster computing with working sets.}, + author={Zaharia, Matei and Chowdhury, Mosharaf and Franklin, Michael J and Shenker, Scott and Stoica, Ion}, + journal={HotCloud}, + volume={10}, + number={10-10}, + pages={95}, + year={2010} +} +@inproceedings{shvachko2010hadoop, + title={The {Hadoop} distributed file system}, + author={Shvachko, Konstantin and Kuang, Hairong and Radia, Sanjay and Chansler, Robert}, + booktitle={IEEE symposium on Mass storage systems and technologies (MSST)}, + pages={1--10}, + year={2010}, + organization={Ieee} +} +@inproceedings{ghoting2011systemml, + title={{SystemML: Declarative machine learning on MapReduce}}, + author={Ghoting, Amol and Krishnamurthy, Rajasekar and Pednault, Edwin and Reinwald, Berthold and Sindhwani, Vikas and Tatikonda, Shirish and Tian, Yuanyuan and Vaithyanathan, Shivakumar}, + booktitle={International Conference on Data Engineering (ICDE)}, + pages={231--242}, + year={2011}, + organization={IEEE} +} +@article{meng2016mllib, + title={Mllib: Machine learning in apache spark}, + author={Meng, Xiangrui and Bradley, Joseph and Yavuz, Burak and Sparks, Evan and Venkataraman, Shivaram and Liu, Davies and Freeman, Jeremy and Tsai, DB and Amde, Manish and Owen, Sean and Doris Xin and Reynold Xin and Michael J. Franklin and Reza Zadeh and Matei Zaharia and Ameet Talwalkar}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={17}, + number={1}, + pages={1235--1241}, + year={2016}, + publisher={JMLR. org} +} +@article{dean2008mapreduce, + title={MapReduce: simplified data processing on large clusters}, + author={Dean, Jeffrey and Ghemawat, Sanjay}, + journal={ACM Communications}, + volume={51}, + number={1}, + pages={107--113}, + year={2008}, + publisher={ACM} +} +@article{curtin2013mlpack, + title={{MLPACK: A scalable C++ machine learning library}}, + author={Curtin, Ryan R and Cline, James R and Slagle, Neil P and March, William B and Ram, Parikshit and Mehta, Nishant A and Gray, Alexander G}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={14}, + number={Mar}, + pages={801--805}, + year={2013} +} +@article{sonnenburg2010shogun, + title={The SHOGUN machine learning toolbox}, + author={Sonnenburg, S{\'C} and Henschel, Sebastian and Widmer, Christian and Behr, Jonas and Zien, Alexander and Bona, Fabio de and Binder, Alexander and Gehl, Christian and Franc, Vojt{\"A} and others}, + journal={Journal of Machine Learning Research (JMLR)}, + volume={11}, + number={Jun}, + pages={1799--1802}, + year={2010} +} + +%% Distributed training algorithm +@inproceedings{mitliagkas2016asynchrony, + title={Asynchrony begets momentum, with an application to deep learning}, + author={Mitliagkas, Ioannis and Zhang, Ce and Hadjis, Stefan and R{\'e}, Christopher}, + booktitle={2016 54th Annual Allerton Conference on Communication, Control, and Computing (Allerton)}, + pages={997--1004}, + year={2016}, + organization={IEEE} +} + +@article{chahal2018hitchhiker, + title={A Hitchhiker's Guide On Distributed Training of Deep Neural Networks}, + author={Chahal, Karanbir and Grover, Manraj Singh and Dey, Kuntal}, + journal={arXiv preprint arXiv:1810.11787}, + year={2018} +} + +@inproceedings{zinkevich2010parallelized, + title={Parallelized stochastic gradient descent}, + author={Zinkevich, Martin and Weimer, Markus and Li, Lihong and Smola, Alex J}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={2595--2603}, + year={2010} +} +@inproceedings{dean2012large, + title={Large scale distributed deep networks}, + author={Dean, Jeffrey and Corrado, Greg and Monga, Rajat and Chen, Kai and Devin, Matthieu and Mao, Mark and Senior, Andrew and Tucker, Paul and Yang, Ke and Le, Quoc V and others}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={1223--1231}, + year={2012} +} +@inproceedings{gupta2016model, + title={Model accuracy and runtime tradeoff in distributed deep learning: A systematic study}, + author={Gupta, Suyog and Zhang, Wei and Wang, Fei}, + booktitle={2016 IEEE 16th International Conference on Data Mining (ICDM)}, + pages={171--180}, + year={2016}, + organization={IEEE} +} + +%% Tools: PS, Horovod +@article{sergeev2018horovod, + title={{Horovod: fast and easy distributed deep learning in TensorFlow}}, + author={Sergeev, Alexander and Del Balso, Mike}, + journal={arXiv preprint arXiv:1802.05799}, + year={2018} +} +@article{goyal2017accurate, + title={Accurate, large minibatch SGD: training imagenet in 1 hour}, + author={Goyal, Priya and Doll{\'a}r, Piotr and Girshick, Ross and Noordhuis, Pieter and Wesolowski, Lukasz and Kyrola, Aapo and Tulloch, Andrew and Jia, Yangqing and He, Kaiming}, + journal={arXiv preprint arXiv:1706.02677}, + year={2017} +} +@inproceedings{li2014scaling, + title={Scaling Distributed Machine Learning with the Parameter Server.}, + author={Li, Mu and Andersen, David G and Park, Jun Woo and Smola, Alexander J and Ahmed, Amr and Josifovski, Vanja and Long, James and Shekita, Eugene J and Su, Bor-Yiing}, + booktitle={Usenix OSDI}, + volume={14}, + pages={583--598}, + year={2014} +} +@article{low2012distributed, + title={Distributed GraphLab: a framework for machine learning and data mining in the cloud}, + author={Low, Yucheng and Bickson, Danny and Gonzalez, Joseph and Guestrin, Carlos and Kyrola, Aapo and Hellerstein, Joseph M}, + journal={Proceedings of the VLDB Endowment}, + volume={5}, + number={8}, + pages={716--727}, + year={2012}, + publisher={VLDB Endowment} +} +@article{xing2015petuum, + title={Petuum: A new platform for distributed machine learning on big data}, + author={Xing, Eric P and Ho, Qirong and Dai, Wei and Kim, Jin Kyu and Wei, Jinliang and Lee, Seunghak and Zheng, Xun and Xie, Pengtao and Kumar, Abhimanu and Yu, Yaoliang}, + journal={IEEE Transactions on Big Data}, + volume={1}, + number={2}, + pages={49--67}, + year={2015}, + publisher={IEEE} +} +@article{chun2013reef, + title={Reef: Retainable evaluator execution framework}, + author={Chun, Byung-Gon and Condie, Tyson and Curino, Carlo and Douglas, Chris and Matusevych, Sergiy and Myers, Brandon and Narayanamurthy, Shravan and Ramakrishnan, Raghu and Rao, Sriram and Rosen, Josh and Russell Sears and Russell Sears}, + journal={Proceedings of the VLDB Endowment}, + volume={6}, + number={12}, + pages={1370--1373}, + year={2013}, + publisher={VLDB Endowment} +} +@inproceedings{murray2013naiad, + title={Naiad: a timely dataflow system}, + author={Murray, Derek G and McSherry, Frank and Isaacs, Rebecca and Isard, Michael and Barham, Paul and Abadi, Mart{\'\i}n}, + booktitle={Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles}, + pages={439--455}, + year={2013}, + organization={ACM} +} +@inproceedings{kraska2013mlbase, + title={MLbase: A Distributed Machine-learning System.}, + author={Kraska, Tim and Talwalkar, Ameet and Duchi, John C and Griffith, Rean and Franklin, Michael J and Jordan, Michael I}, + booktitle={Cidr}, + volume={1}, + pages={2--1}, + year={2013} +} +@misc{baidu2017ringallreduce, + author = {Andrew Gibiansky}, + title = {Bringing HPC techniques to deep learning.}, + year = {2017}, + howpublished = {\url{http://research.baidu.com/bringing-hpc-techniques-deep-learning}} +} + +%%% Tools : Theano TensorFlow + +@inproceedings{abadi2016tensorflow, + title={TensorFlow: A system for large-scale machine learning}, + author={Abadi, Martin and Barham, Paul and Chen, Jianmin and Davis, Andy and Dean, Jeffrey and Devin, Matthieu and Geoffrey, Sanjay and Irving, Geoffrey and Devin, Michael and Kudlur, Manjunath and Manjunath, Josh and Monga, Rajat and Moore, Sherry and G. Murray, Derek and Derek, Benoit and Tucker, Paul and Vasudevan, Vijay and Warden, Pete and Wicke, Martin and Yu, Yuan and Xiaoqiang Zheng + }, + booktitle={Usenix OSDI}, + year={2016} +} + +@inproceedings{seide2016cntk, + title={{CNTK: Microsoft's} Open-Source Deep-Learning Toolkit}, + author={Seide, Frank and Agarwal, Amit}, + booktitle={Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD)}, + year={2016} +} + +@article{xiao2020thinking, + title={Thinking While Moving: Deep Reinforcement Learning with Concurrent Control}, + author={Xiao, Ted and Jang, Eric and Kalashnikov, Dmitry and Levine, Sergey and Ibarz, Julian and Hausman, Karol and Herzog, Alexander}, + journal={arXiv preprint arXiv:2004.06089}, + year={2020} +} + + +@inproceedings{chen2015mxnet, + title={Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems}, + author={Tianqi Chen and Mu Li and Yutian Li and Min Lin and Naiyan Wang and Minjie Wang and Tianjun Xiao and Bing Xu and Chiyuan Zhang and Zheng Zhang}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop}, + year={2015} +} + +@inproceedings{jia2014caffe, + title={Caffe: Convolutional architecture for fast feature embedding}, + author={Jia, Yangqing and Shelhamer, Evan and Donahue, Jeff and Sergey Karayev and Jonathan Long and Ross Girshick and Sergio Guadarrama and Trevor Darrell}, + booktitle={ACM Multimedia}, + year={2014} +} + +@techreport{bergstra2010theano, + title={{Theano: A CPU and GPU math compiler in Python}}, + author={Bergstra, James and Breuleux, Olivier and others}, + year={2010} +} + +@techreport{collobert2002torch, + title={Torch: a modular machine learning software library}, + author={Collobert, Ronan and Bengio, Samy and Mari{\'e}thoz, Johnny}, + year={2002}, + institution={Technical Report IDIAP-RR 02-46, IDIAP} +} + +@misc{sonnet, + author = {Deepmind}, + title = {Sonnet}, + year = {2017}, + howpublished = {\url{https://github.com/deepmind/sonnet}} +} + +@misc{stanforddlsoftware, + author = {Fei-Fei, Li and Justin, Johnson and Serena, Yeung}, + title = {Stanford : Deep Learning Software}, + year = {2017}, + howpublished = {\url{http://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture8.pdf}} +} + +@inproceedings{paszke2017pytorch, + title={Automatic differentiation in PyTorch}, + author={Paszke, Adam and Gross, Sam and Chintala, Soumith and Chanan, Gregory and Yang, Edward and DeVito, Zachary and Lin, Zeming and Desmaison, Alban and Antiga, Luca and Lerer, Adam}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop}, + year={2017} +} + +@misc{tflearn2016xxx, + author = {Aymeric Damien}, + title = {TFLearn: Deep learning library featuring a higher-level API for TensorFlow}, + year = {2016} , + publisher={GitHub}, + howpublished={\url{https://github.com/tflearn/tflearn}, \url{http://tflearn.org}}, +} + +@misc{chollet2015xxx, + author = {François Chollet}, + title = {Keras: Deep Learning for humans}, + year = {2015}, + publisher = {GitHub}, + journal = {GitHub repository}, + howpublished = {\url{https://github.com/fchollet/keras}, \url{http://keras.io}}, + commit = {5bcac37} +} + +@misc{benchmark, + author = {TensorFlow}, + title = {TensorFlow Tutorials}, + year = {2017}, + howpublished = {\url{https://www.tensorflow.org/tutorials/}} +} + +@misc{lasagne, + author = {Sander Dieleman and + Jan Schlüter and + others}, + title = {Lasagne}, + year = {2015}, + howpublished = {\url{https://github.com/Lasagne/Lasagne}} +} + +@misc{dlranking2017, + author={{GithubReviews Team}}, + title = {Machine Learning Repository Review Ranking}, + year = {2017}, + howpublished={\url{https://githubreviews.com/explore/machine-learning},} +} + + +@inproceedings{krogh1995neural, + title={Neural network ensembles, cross validation, and active learning}, + author={Krogh, Anders and Vedelsby, Jesper and others}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={1995}, +} + + +@inproceedings{mikolov2010recurrent, + title={Recurrent neural network based language model}, + author={Mikolov, Tomas and Karafi{\'a}t, Martin and Lukas Burget and Jan Cernocky and Sanjeev Khudanpur }, + booktitle={Interspeech}, + year={2010} +} +%% binarynet +@article{courbariaux2016binarized, + title={Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1}, + author={Courbariaux, Matthieu and Hubara, Itay and Soudry, Daniel and El-Yaniv, Ran and Bengio, Yoshua}, + journal={CoRR}, + year={2016} +} +@inproceedings{li2016ternary, + booktitle = {Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + title = {{Ternary Weight Networks}}, + author={Fengfu Li and Bo Zhang and Bin Liu}, + year = {2016} +} +@inproceedings{zhou2016dorefa, + title={Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients}, + author={Zhou, Shuchang and Wu, Yuxin and Ni, Zekun and Zhou, Xinyu and Wen, He and Zou, Yuheng}, + booktitle={CoRR}, + year={2016} +} +@article{hubara2017quantized, + title={Quantized neural networks: Training neural networks with low precision weights and activations}, + author={Hubara, Itay and Courbariaux, Matthieu and Soudry, Daniel and El-Yaniv, Ran and Bengio, Yoshua}, + journal={The Journal of Machine Learning Research (JMLR)}, + volume={18}, + number={1}, + pages={6869--6898}, + year={2017}, +} + +%% computer graphic related / learn from synthesised data +@inproceedings{peng2018visda, + title={VisDA: A Synthetic-to-Real Benchmark for Visual Domain Adaptation}, + author={Peng, Xingchao and Usman, Ben and Kaushik, Neela and Wang, Dequan and Hoffman, Judy and Saenko, Kate}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops}, + pages={2021--2026}, + year={2018} +} +@inproceedings{chen2016synthesizing, + title={Synthesizing training images for boosting human 3d pose estimation}, + author={Chen, Wenzheng and Wang, Huan and Li, Yangyan and Su, Hao and Wang, Zhenhua and Tu, Changhe and Lischinski, Dani and Cohen-Or, Daniel and Chen, Baoquan}, + booktitle={Proceedings of the International Conference on 3D Vision (3DV)}, + pages={479--488}, + year={2016}, + organization={IEEE} +} +@inproceedings{parish2001procedural, + title={Procedural modeling of cities}, + author={Parish, Yoav IH and M{\"u}ller, Pascal}, + booktitle={Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques}, + pages={301--308}, + year={2001}, +} +@inproceedings{frid2018synthetic, + title={Synthetic data augmentation using GAN for improved liver lesion classification}, + author={Frid-Adar, Maayan and Klang, Eyal and Amitai, Michal and Goldberger, Jacob and Greenspan, Hayit}, + booktitle={International Symposium on Biomedical Imaging (ISBI)}, + pages={289--293}, + year={2018}, + organization={IEEE} +} +@article{antoniou2017data, + title={Data augmentation generative adversarial networks}, + author={Antoniou, Antreas and Storkey, Amos and Edwards, Harrison}, + journal={arXiv preprint arXiv:1711.04340}, + year={2017} +} +@article{frid2018gan, + title={GAN-based synthetic medical image augmentation for increased CNN performance in liver lesion classification}, + author={Frid-Adar, Maayan and Diamant, Idit and Klang, Eyal and Amitai, Michal and Goldberger, Jacob and Greenspan, Hayit}, + journal={Neurocomputing}, + volume={321}, + pages={321--331}, + year={2018}, + publisher={Elsevier} +} + +%% interactive active learning +@article{amershi2014power, + title={Power to the people: The role of humans in interactive machine learning}, + author={Amershi, Saleema and Cakmak, Maya and Knox, William Bradley and Kulesza, Todd}, + journal={AI Magazine}, + year={2014} +} + +@article{gal2017deep, + title={Deep Bayesian Active Learning with Image Data}, + author={Gal, Yarin and Islam, Riashat and Ghahramani, Zoubin}, + journal={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} + +%% hyper-parameter selection +@article{berg2012random, + title={Random search for hyper-parameter optimization}, + author={Bergstra, James and Bengio, Yoshua}, + journal={Journal of Machine Learning Research (JMLR)}, + year={2012} +} +@inproceedings{jiang2017interactive, + title={Interactive Machine Learning via a GPU-accelerated Toolkit}, + author={Jiang, Biye and Canny, John}, + booktitle={Annual meeting of the ACM Intelligent User Interface (IUI)}, + year={2017} +} % no deep learning + +%% DL management +@inproceedings{miao2017modelhub, + title={ModelHub: Deep Learning Lifecycle Management}, + author={Miao, Hui and Li, Ang and Davis, Larry S and Deshpande, Amol}, + booktitle={International Conference on Data Engineering (ICDE)}, + year={2017} +} + +@inproceedings{andreas2016nmn, + title={Neural module networks}, + author={Andreas, Jacob and Rohrbach, Marcus and Darrell, Trevor and Klein, Dan}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year={2016} +} + +@inproceedings{akdere2011case, + title={The Case for Predictive Database Systems: Opportunities and Challenges.}, + author={Akdere, Mert and Cetintemel, Ugur and Riondato, Matteo and Upfal, Eli and Zdonik, Stanley B}, + booktitle={the Biennial Conference on Innovative Data Systems Research (CIDR)}, + pages={167--174}, + year={2011} +} +@inproceedings{feng2012towards, + title={Towards a unified architecture for in-RDBMS analytics}, + author={Feng, Xixuan and Kumar, Arun and Recht, Benjamin and R{\'e}, Christopher}, + booktitle={Proceedings of the ACM SIGMOD International Conference on Management of Data (SIGMOD)}, + pages={325--336}, + year={2012}, + organization={ACM} +} +@inproceedings{vartak2015sherlock, + title={Supporting fast iteration in model building}, + author={Vartak, Manasi and Ortiz, Pablo and Siegel, Kathryn and Subramanyam, Harihar and Madden, Samuel and Zaharia, Matei}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop of Learning System}, + year={2015} +} +@article{hines2004modeldb, + title={{ModelDB: a database to support computational neuroscience}}, + author={Hines, Michael L and Morse, Thomas and Migliore, Michele and Carnevale, Nicholas T and Shepherd, Gordon M}, + journal={Journal of Computational Neuroscience}, + volume={17}, + number={1}, + pages={7--11}, + year={2004}, + publisher={Springer} +} +@inproceedings{callahan2006vistrails, + title={VisTrails: visualization meets data management}, + author={Callahan, Steven P and Freire, Juliana and Santos, Emanuele and Scheidegger, Carlos E and Silva, Cl{\'a}udio T and Vo, Huy T}, + booktitle={ACM SIGMOD international conference on Management of data}, + pages={745--747}, + year={2006}, + organization={ACM} +} +@article{ludascher2006kepler, + title={Scientific workflow management and the Kepler system}, + author={Lud{\"a}scher, Bertram and Altintas, Ilkay and Berkley, Chad and Higgins, Dan and Jaeger, Efrat and Jones, Matthew and Lee, Edward A and Tao, Jing and Zhao, Yang}, + journal={Concurrency and Computation: Practice and Experience}, + volume={18}, + number={10}, + pages={1039--1065}, + year={2006}, + publisher={Wiley Online Library} +} + +@inproceedings{johannink2019residual, + title={Residual reinforcement learning for robot control}, + author={Johannink, Tobias and Bahl, Shikhar and Nair, Ashvin and Luo, Jianlan and Kumar, Avinash and Loskyll, Matthias and Ojea, Juan Aparicio and Solowjow, Eugen and Levine, Sergey}, + booktitle={2019 International Conference on Robotics and Automation (ICRA)}, + pages={6023--6029}, + year={2019}, + organization={IEEE} +} + +@inproceedings{hansen1996adapting, + title={Adapting arbitrary normal mutation distributions in evolution strategies: The covariance matrix adaptation}, + author={Hansen, Nikolaus and Ostermeier, Andreas}, + booktitle={Proceedings of IEEE international conference on evolutionary computation}, + pages={312--317}, + year={1996}, + organization={IEEE} +} + + +@article{nelder1965simplex, + title={A simplex method for function minimization}, + author={Nelder, John A and Mead, Roger}, + journal={The computer journal}, + volume={7}, + number={4}, + pages={308--313}, + year={1965}, + publisher={Oxford University Press} +} + + + +@article{wolstencroft2013taverna, + title={The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud}, + author={Wolstencroft, Katherine and Haines, Robert and Fellows, Donal and Williams, Alan and Withers, David and Owen, Stuart and Soiland-Reyes, Stian and Dunlop, Ian and Nenadic, Aleksandra and Fisher, Paul and others}, + journal={Nucleic acids research}, + volume={41}, + number={W1}, + pages={W557--W561}, + year={2013}, + publisher={Oxford University Press} +} + +%% quantised binary net +@inproceedings{courbariaux2016binarized, + title={Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1}, + author={Courbariaux, Matthieu and Hubara, Itay and Soudry, Daniel and El-Yaniv, Ran and Bengio, Yoshua}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year={2016} +} +@inproceedings{li2016ternary, + title={Ternary Weight Networks}, + author={Li, Fengfu and Zhang, Bo and Liu, Bin}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop}, + year={2016} +} +@article{zhou2016dorefa, + title={Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients}, + author={Zhou, Shuchang and Wu, Yuxin and Ni, Zekun and Zhou, Xinyu and Wen, He and Zou, Yuheng}, + journal={arXiv preprint arXiv:1606.06160}, + year={2016} +} + +@inproceedings{jaderberg2015spatial, + title={Spatial transformer networks}, + author={Max Jaderberg and Karen Simonyan and Andrew Zisserman and Koray Kavukcuoglu}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={2017--2025}, + year={2015} +} + +@article{silver2018residual, + title={Residual policy learning}, + author={Silver, Tom and Allen, Kelsey and Tenenbaum, Josh and Kaelbling, Leslie}, + journal={arXiv preprint arXiv:1812.06298}, + year={2018} +} + + +%% Alexnet +@inproceedings{krizhevsky2012alexnet, + title={Imagenet classification with deep convolutional neural networks}, + author={Krizhevsky, Alex and Sutskever, Ilya and Hinton, Geoffrey E}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems)}, + pages={1097--1105}, + year={2012} +} + +%%% VGG +@inproceedings{Simonyan2015, + title={{Very deep convolutional networks for large-scale image recognition}}, + author={Simonyan, Karen and Zisserman, Andrew}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2015}, +} +@article{howard2017mobilenets, + title={Mobilenets: Efficient convolutional neural networks for mobile vision applications}, + author={Howard, Andrew G and Zhu, Menglong and Chen, Bo and Kalenichenko, Dmitry and Wang, Weijun and Weyand, Tobias and Andreetto, Marco and Adam, Hartwig}, + journal={Computing Research Repository (CoRR)}, + year={2017} +} +@article{iandola2016squeezenet, + title={Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size}, + author={Iandola, Forrest N and Han, Song and Moskewicz, Matthew W and Ashraf, Khalid and Dally, William J and Keutzer, Kurt}, + journal={arXiv preprint arXiv:1602.07360}, + year={2016} +} +@misc{vggarchitecture, + title = {Build Deeper: Deep Learning Beginners' Guide}, + author = {Codes of Interest Blog}, + url = {{https://www.codesofinterest.com/p/build-deeper.html}}, +} + +@inproceedings{he2017channel, + title={Channel pruning for accelerating very deep neural networks}, + author={He, Yihui and Zhang, Xiangyu and Sun, Jian}, + booktitle = {Proceedings of the International Conference on Computer Vision (ICCV)}, + volume={2}, + number={6}, + year={2017} +} + +@inproceedings{Ioffe2015, + title={{Batch normalization: accelerating deep network training by reducing internal covariate shift}}, + author={Ioffe, Sergey and Szegedy, Christian}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2015}, +} +% pix2pix +@inproceedings{Isola2017, + author = {Isola, Phillip and Zhu, Jun-Yan and Zhou, Tinghui and Efros, Alexei A.}, + title = {{Image-to-image translation with conditional adversarial networks}}, + year = {2017}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, +} + +% bug +@inproceedings{chen2017iccv, + title = {{Photographic image synthesis with cascaded refinement networks}}, + author = {Chen, Qifeng and Vladlen, Koltun}, + booktitle = {Proceedings of International Conference on Computer Vision (ICCV)}, + year = {2017}, +} + +% pix2pixhd % bug +@inproceedings{wang2018cvpr, + title = {{High-resolution image synthesis and semantic manipulation with conditional GANs}}, + author = {Ting-Chun, Wang and Ming-Yu, Li and Jun-Yan, Zhu and Andrew, Tao and Jan, Kautz and Bryan, Catanzaro}, + booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + year = {2018}, +} + +@article{Eksioglu2016, + title={Decoupled algorithm for {MRI} reconstruction using nonlocal block matching model: {BM3D-MRI}}, + author={Eksioglu, Ender M}, + journal={Journal of Mathematical Imaging and Vision}, + volume={56}, + number={3}, + pages={430--440}, + year={2016}, + publisher={Springer} +} +@article{Zhan2016, + title={Fast multiclass dictionaries learning with geometrical directions in {MRI} reconstruction}, + author={Zhan, Zhifang and Cai, Jian-Feng and Guo, Di and Liu, Yunsong and Chen, Zhong and Qu, Xiaobo}, + journal={IEEE Transactions on Biomedical Engineering}, + volume={63}, + number={9}, + pages={1850--1861}, + year={2016}, + publisher={IEEE} +} +@inproceedings{KingmaAdam2014, + title = {{Adam: a method for stochastic optimization}}, + author = {Kingma, Diederik and Ba, Jimmy}, + booktitle = {Proceedings of the International Conference on Learning Representations (ICLR)}, + year = {2014} +} +@article{Mirza2014, + author = {Mirza, Mehdi and Osindero, Simon}, + title = {{Conditional generative adversarial nets}}, + year = {2014}, + journal = {arXiv preprint arXiv:1411.1784}, +} + +@inproceedings{Goodfellow2014, + title = {Generative adversarial nets}, + author = {Goodfellow, Ian J and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil and Courville, Aaron and Bengio, Yoshua}, + booktitle = {Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year = {2014}, +} + +@book{priemer1990introductory, +title={Introductory signal processing}, +author={Priemer, Roland}, +volume={6}, +year={1990}, +publisher={World Scientific Publishing Company} +} +@inproceedings{Agostinelli2013, + title = {Adaptive multi-column deep neural networks with application to robust image denoising}, + author = {Agostinelli, Forest and Anderson, Michael R and Lee, Honglak}, + booktitle = {Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year = {2013}, +} +@inproceedings{Xie2012, + title = {Image denoising and inpainting with deep neural networks}, + author = {Xie, Junyuan and Xu, Linli and Chen, Enhong}, + booktitle = {Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + year = {2012}, +} +@article{Dong2016, + author = {Dong, Chao and Loy, Chen Change and He, Kaiming and Tang, Xiaoou}, + doi = {10.1109/TPAMI.2015.2439281}, + issn = {0162-8828}, + journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI)}, + number = {2}, + pages = {295--307}, + title = {{Image super-resolution using deep convolutional networks}}, + url = {http://ieeexplore.ieee.org/document/7115171/}, + volume = {38}, + year = {2016} +} + +@article{Lee2017, + author = {Lee, Dongwook and Yoo, Jaejun and Ye, Jong Chul}, + title = {{Deep artifact learning for compressed sensing and parallel MRI}}, + year = {2017}, + journal = {arXiv preprint arXiv:1703.01120}, +} +@article{Hammernik2017, + author = {Hammernik, Kerstin and Klatzer, Teresa and Kobler, Erich and Recht, Michael P and Sodickson, Daniel K and Pock, Thomas and Knoll, Florian}, + title = {{Learning a variational network for reconstruction of accelerated MRI data}}, + year = {2017}, + journal = {arXiv preprint arXiv:1704.00447}, +} +@article{Schlemper2017a, + author = {Schlemper, Jo and Caballero, Jose and Hajnal, Joseph V. and Price, Anthony and Rueckert, Daniel}, + title = {{A deep cascade of convolutional neural networks for dynamic MR image reconstruction}}, + year = {2017}, + journal = {arXiv preprint arXiv:1704.02422}, +} +@inproceedings{Wang2016, + author = {Wang, Shanshan and Su, Zhenghang and Ying, Leslie and Peng, Xi and Zhu, Shun and Liang, Feng and Feng, Dagan and Liang, Dong}, + booktitle = {IEEE International Symposium on Biomedical Imaging (ISBI)}, + title = {{Accelerating magnetic resonance imaging via deep learning}}, + year = {2016} +} +@inproceedings{Yang2016, + author = {Yang, Yan and Sun, Jian and Li, Huibin and Xu, Zongben}, + booktitle = {Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + title = {{Deep ADMM-Net for compressive sensing MRI}}, + year = {2016} +} +@article{Jaspan2015, + author = {Jaspan, Oren N. and Fleysher, Roman and Lipton, Michael L.}, + doi = {10.1259/bjr.20150487}, + issn = {00071285}, + journal = {The British Journal of Radiology}, + number = {1056}, + pages = {1--12}, + pmid = {26402216}, + title = {{Compressed sensing MRI: a review of the clinical literature}}, + volume = {88}, + year = {2015} +} +@article{kleesiek2016deep, + title={Deep MRI brain extraction: a 3D convolutional neural network for skull stripping}, + author={Kleesiek, Jens and Urban, Gregor and Hubert, Alexander and Schwarz, Daniel and Maier-Hein, Klaus and Bendszus, Martin and Biller, Armin}, + journal={NeuroImage}, + volume={129}, + pages={460--469}, + year={2016}, + publisher={Elsevier} +} +@article{milletari2017hough, + title={Hough-CNN: deep learning for segmentation of deep brain regions in MRI and ultrasound}, + author={Milletari, Fausto and Ahmadi, Seyed-Ahmad and Kroll, Christine and Plate, Annika and Rozanski, Verena and Maiostre, Juliana and Levin, Johannes and Dietrich, Olaf and Ertl-Wagner, Birgit and B{\"o}tzel, Kai and others}, + journal={Computer Vision and Image Understanding}, + volume={164}, + pages={92--102}, + year={2017}, + publisher={Elsevier} +} +@inproceedings{cciccek20163d, + title={3D U-Net: learning dense volumetric segmentation from sparse annotation}, + author={{\c{C}}i{\c{c}}ek, {\"O}zg{\"u}n and Abdulkadir, Ahmed and Lienkamp, Soeren S and Brox, Thomas and Ronneberger, Olaf}, + booktitle={Proceedings of the International Conference On Medical Image Computing \& Computer Assisted Intervention (MICCAI)}, + pages={424--432}, + year={2016}, + organization={Springer} +} +@article{Knoll2012, + author = {Knoll, Florian and Clason, Christian and Bredies, Kristian and Uecker, Martin and Stollberger, Rudolf}, + doi = {10.1002/mrm.22964}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine}, + number = {1}, + pages = {34--41}, + title = {{Parallel imaging with nonlinear reconstruction using variational penalties}}, + url = {http://doi.wiley.com/10.1002/mrm.22964}, + volume = {67}, + year = {2012} +} +@article{Knoll2011, + author = {Knoll, Florian and Bredies, Kristian and Pock, Thomas and Stollberger, Rudolf}, + doi = {10.1002/mrm.22595}, + issn = {1522-2594}, + journal = {Magnetic Resonance in Medicine}, + number = {2}, + pages = {480--491}, + pmid = {21264937}, + title = {{Second order total generalized variation (TGV) for MRI}}, + url = {http://www.ncbi.nlm.nih.gov/pubmed/21264937 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4011128}, + volume = {65}, + year = {2011} +} +@inproceedings{Ronneberger2015, + author = {Ronneberger, Olaf and Fischer, Philipp and Brox, Thomas}, + booktitle = {Proceedings of the International Conference On Medical Image Computing \& Computer Assisted Intervention (MICCAI)}, + title = {{U-Net: convolutional networks for biomedical image segmentation}}, + year = {2015} +} +@inproceedings{milletari2016vnet, + title={V-net: Fully convolutional neural networks for volumetric medical image segmentation}, + author={Milletari, Fausto and Navab, Nassir and Ahmadi, Seyed-Ahmad}, + booktitle={Proceedings of the International Conference on 3D Vision (3DV)}, + pages={565--571}, + year={2016}, + organization={IEEE} +} +@article{Zhu2013, + author = {Zhu, Zangen and Wahid, Khan and Babyn, Paul and Yang, Ran}, + doi = {10.1155/2013/907501}, + isbn = {1687-4188 (Print) 1687-4188 (Linking)}, + issn = {1687-4188}, + journal = {International Journal of Biomedical Imaging}, + pages = {10}, + pmid = {23840199}, + title = {{Compressed sensing-based MRI reconstruction using complex double-density dual-tree DWT}}, + volume = {2013}, + year = {2013} +} +@article{Fair2015, + author = {Fair, Merlin J and Gatehouse, Peter D and DiBella, Edward V R and Firmin, David N}, + doi = {10.1186/s12968-015-0162-9}, + issn = {1532-429X}, + journal = {J. Cardiovasc. Magn. Reson.}, + keywords = {3d,Myocardial perfusion,3D,Whole heart,Cardiovascular,cardiovascular magnetic resonance,myocardial perfusion,whole heart}, + number = {1}, + pages = {68}, + pmid = {26231784}, + publisher = {Journal of Cardiovascular Magnetic Resonance}, + title = {{A review of 3D first-pass, whole-heart, myocardial perfusion cardiovascular magnetic resonance}}, + volume = {17}, + year = {2015} +} +@article{Strohmer2012, + archivePrefix = {arXiv}, + arxivId = {arXiv:1210.6730v1}, + author = {Strohmer, Thomas}, + doi = {10.1109/LSP.2012.2224518}, + eprint = {arXiv:1210.6730v1}, + isbn = {1070-9908}, + issn = {1070-9908}, + journal = {IEEE Signal Processing Letters}, + month = {dec}, + number = {12}, + pages = {887--893}, + title = {{Measure What Should be Measured: Progress and Challenges in Compressive Sensing}}, + url = {http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=6338308}, + volume = {19}, + year = {2012} +} +@article{Ning2013, + author = {Ning, Bende and Qu, Xiaobo and Guo, Di and Hu, Changwei and Chen, Zhong}, + doi = {10.1016/j.mri.2013.07.010}, + issn = {0730725X}, + journal = {Journal of Magnetic Resonance Imaging (JMRI)}, + month = {nov}, + number = {9}, + pages = {1611--1622}, + title = {{Magnetic resonance image reconstruction using trained geometric directions in 2D redundant wavelets domain and non-convex optimization}}, + url = {http://linkinghub.elsevier.com/retrieve/pii/S0730725X13002555}, + volume = {31}, + year = {2013} +} +@article{Yang2015, + author = {Yang, Bingxin and Yuan, Min and Ma, Yide and Zhang, Jiuwen and Zhan, Kun}, + doi = {10.1186/s12880-015-0065-0}, + issn = {1471-2342}, + journal = {BMC medical imaging}, + number = {1}, + pages = {28}, + pmid = {26253135}, + publisher = {BMC Medical Imaging}, + title = {{Local sparsity enhanced compressed sensing magnetic resonance imaging in uniform discrete curvelet domain.}}, + url = {http://www.biomedcentral.com/1471-2342/15/28 http://www.ncbi.nlm.nih.gov/pubmed/26253135 http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC4528851}, + volume = {15}, + year = {2015} +} +@article{JunfengYang2010, + author = {Yang, Junfeng and Zhang, Yin and Yin, Wotao}, + doi = {10.1109/JSTSP.2010.2042333}, + issn = {1932-4553}, + journal = {IEEE Journal of Selected Topics in Signal Processing}, + number = {2}, + pages = {288--297}, + title = {{A fast alternating direction method for TVL1-L2 signal reconstruction from partial fourier data}}, + volume = {4}, + year = {2010} +} +@article{Qu2014, + author = {Qu, Xiaobo and Hou, Yingkun and Lam, Fan and Guo, Di and Zhong, Jianhui and Chen, Zhong}, + doi = {10.1016/j.media.2013.09.007}, + issn = {13618423}, + journal = {Medical Image Analysis}, + keywords = {Compressed sensing,Image reconstruction,Magnetic resonance imaging,Nonlocal operator,Sparsity}, + number = {6}, + pages = {843--856}, + pmid = {24176973}, + publisher = {Elsevier B.V.}, + title = {{Magnetic resonance image reconstruction from undersampled measurements using a patch-based nonlocal operator}}, + url = {http://dx.doi.org/10.1016/j.media.2013.09.007}, + volume = {18}, + year = {2014} +} +@article{Gdaniec2014, + title={Robust abdominal imaging with incomplete breath-holds}, + author={Gdaniec, Nadine and Eggers, Holger and B{\"o}rnert, Peter and Doneva, Mariya and Mertins, Alfred}, + journal={Magnetic resonance in medicine}, + volume={71}, + number={5}, + pages={1733--1742}, + year={2014}, + publisher={Wiley Online Library} +} + +@article{Ravishankar2011, + author = {Ravishankar, Saiprasad and Bresler, Yoram}, + doi = {10.1109/TMI.2010.2090538}, + file = {:privateUsers/gyang/Dropbox/Papers/CompressiveSensing/Ravishankar11{\_}HighlySparseCSUsingDictionaryLearning.pdf:pdf}, + isbn = {0278-0062}, + issn = {1558-254X}, + journal = {IEEE Transactions on Medical Imaging (TMI)}, + keywords = {Compressed sensing (CS),dictionary learning,image reconstruction,magnetic resonance imaging (MRI),reduced encoding,sparse representation}, + number = {5}, + pages = {1028--1041}, + pmid = {21047708}, + title = {{MR image reconstruction from highly undersampled k-space data by dictionary learning}}, + url = {http://www.ncbi.nlm.nih.gov/pubmed/21047708}, + volume = {30}, + year = {2011} +} +@inproceedings{Wang2014, + author = {Wang, Yanhua and Ying, Leslie}, + doi = {10.1109/EMBC.2014.6943894}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Wang, Ying - 2014 - Undersampled dynamic magnetic resonance imaging using kernel principal component analysis.pdf:pdf}, + issn = {1557-170X}, + booktitle = {Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)}, + pmid = {25570262}, + title = {{Undersampled dynamic magnetic resonance imaging using kernel principal component analysis}}, + url = {http://www.ncbi.nlm.nih.gov/pubmed/25570262}, + year = {2014} +} +@article{Vasanawala2010, + author = {Vasanawala, Shreyas S and Alley, Marcus T and Hargreaves, Brian A and Barth, Richard A and Pauly, John M and Lustig, Michael}, + issn = {0033-8419}, + journal = {Radiology}, + month = {aug}, + number = {2}, + pages = {607--616}, + title = {{Improved Pediatric MR Imaging with Compressed Sensing}}, + url = {http://pubs.rsna.org/doi/abs/10.1148/radiol.10091218}, + volume = {256}, + year = {2010} +} +@article{KameshIyer2016, + author = {{Kamesh Iyer}, Srikant and Tasdizen, Tolga and Burgon, Nathan and Kholmovski, Eugene and Marrouche, Nassir and Adluru, Ganesh and DiBella, Edward}, + doi = {10.1016/j.mri.2016.03.002}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Kamesh Iyer et al. - 2016 - Compressed sensing for rapid late gadolinium enhanced imaging of the left atrium A preliminary study.pdf:pdf}, + issn = {18735894}, + journal = {Journal of Magnetic Resonance Imaging (JMRI)}, + keywords = {Atrial fibrillation,Compressed sensing,Fast minimization,LGE imaging of the left atrium,MRI}, + number = {7}, + pages = {846--854}, + pmid = {26968143}, + publisher = {Elsevier Inc.}, + title = {{Compressed sensing for rapid late gadolinium enhanced imaging of the left atrium: A preliminary study}}, + url = {http://dx.doi.org/10.1016/j.mri.2016.03.002}, + volume = {34}, + year = {2016} +} +@article{Lustig2010, + archivePrefix = {arXiv}, + arxivId = {NIHMS150003}, + author = {Lustig, Michael and Pauly, John M.}, + doi = {10.1002/mrm.22428}, + eprint = {NIHMS150003}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Lustig, Pauly - 2010 - SPIRiT Iterative self-consistent parallel imaging reconstruction from arbitrary k-space.pdf:pdf}, + isbn = {0740-3194}, + issn = {1522-2594}, + journal = {Magnetic Resonance in Medicine}, + keywords = {Algorithms,Computer-Assisted,Computer-Assisted: methods,Humans,Image Enhancement,Image Enhancement: methods,Image Interpretation,Imaging,Magnetic Resonance Imaging,Magnetic Resonance Imaging: instrumentation,Magnetic Resonance Imaging: methods,Phantoms,Reproducibility of Results,Sensitivity and Specificity}, + month = {aug}, + number = {2}, + pages = {457--71}, + pmid = {20665790}, + title = {{SPIRiT: Iterative self-consistent parallel imaging reconstruction from arbitrary k-space.}}, + volume = {64}, + year = {2010} +} +@article{Seeger2010, + author = {Seeger, Matthias and Nickisch, Hannes and Pohmann, Rolf and Sch{\"{o}}lkopf, Bernhard}, + doi = {10.1002/mrm.22180}, + issn = {1522-2594}, + journal = {Magnetic Resonance in Medicine}, + month = {jan}, + number = {1}, + pages = {116--26}, + pmid = {19859957}, + title = {{Optimization of k-space trajectories for compressed sensing by Bayesian experimental design.}}, + url = {http://www.ncbi.nlm.nih.gov/pubmed/19859957}, + volume = {63}, + year = {2010} +} +@article{Knoll2015, + author = {Knoll, Florian and Raya, Jos{\'{e}} G and Halloran, Rafael O and Baete, Steven and Sigmund, Eric and Bammer, Roland and Block, Tobias and Otazo, Ricardo and Sodickson, Daniel K}, + doi = {10.1002/nbm.3258}, + isbn = {1099-1492 (Electronic)$\backslash$r0952-3480 (Linking)}, + issn = {09523480}, + journal = {NMR in Biomedicine}, + month = {mar}, + number = {3}, + pages = {353--366}, + pmid = {25594167}, + title = {{A model-based reconstruction for undersampled radial spin-echo DTI with variational penalties on the diffusion tensor}}, + url = {http://doi.wiley.com/10.1002/nbm.3258}, + volume = {28}, + year = {2015} +} +@article{Huang2015, + author = {Huang, Xiaolin and Liu, Yipeng and Shi, Lei and {Van Huffel}, Sabine and a.K. Suykens, Johan}, + doi = {10.1016/j.sigpro.2014.09.028}, + issn = {01651684}, + journal = {Signal Processing}, + number = {March 2015}, + pages = {459--475}, + publisher = {Elsevier}, + title = {{Two-level ℓ1 minimization for compressed sensing}}, + url = {http://dx.doi.org/10.1016/j.sigpro.2014.09.028}, + volume = {108}, + year = {2015} +} +@article{Candes2006, + author = {Candes, Emmanuel J. and Tao, Terence}, + doi = {10.1109/TIT.2006.885507}, + issn = {0018-9448}, + journal = {IEEE Transactions on Information Theory}, + month = {dec}, + number = {12}, + pages = {5406--5425}, + title = {{Near-Optimal Signal Recovery From Random Projections: Universal Encoding Strategies?}}, + volume = {52}, + year = {2006} +} +@article{Iyer2016, + author = {Iyer, Srikant Kamesh and Tasdizen, Tolga and Burgon, Nathan and Kholmovski, Eugene and Marrouche, Nassir and Adluru, Ganesh and DiBella, Edward}, + doi = {10.1016/j.mri.2016.03.002}, + issn = {1873-5894}, + journal = {Journal of Magnetic Resonance Imaging (JMRI)}, + month = {mar}, + pmid = {26968143}, + title = {{Compressed sensing for rapid late gadolinium enhanced imaging of the left atrium: A preliminary study.}}, + year = {2016} +} +@article{Nyquist1928, + author = {Nyquist, H.}, + doi = {10.1109/T-AIEE.1928.5055024}, + issn = {0096-3860}, + journal = {IEEE Transactions of the American Institute of Electrical Engineers}, + number = {2}, + pages = {617--644}, + title = {{Certain topics in telegraph transmission theory}}, + volume = {47}, + year = {1928} +} +@article{Duarte-Carvajalino2009, + author = {Duarte-Carvajalino, Julio Martin and Sapiro, Guillermo}, + doi = {10.1109/TIP.2009.2022459}, + issn = {1057-7149}, + journal = {IEEE Transactions on Image Processing (TIP)}, + month = {jul}, + number = {7}, + pages = {1395--408}, + pmid = {19497818}, + title = {{Learning to sense sparse signals: simultaneous sensing matrix and sparsifying dictionary optimization.}}, + volume = {18}, + year = {2009} +} +@article{Lustig2007, + author = {Lustig, Michael and Donoho, David and Pauly, John M.}, + doi = {10.1002/mrm.21391}, + file = {:privateUsers/gyang/Dropbox/Papers/CompressiveSensing/Lustig07{\_}SparseMRI.pdf:pdf}, + isbn = {0740-3194}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine }, + number = {6}, + pages = {1182--1195}, + pmid = {17969013}, + title = {{Sparse MRI: the application of compressed sensing for rapid MR imaging}}, + volume = {58}, + year = {2007} +} +@article{Trzasko2009, + author = {Trzasko, Joshua and Manduca, Armando}, + doi = {10.1109/TMI.2008.927346}, + issn = {0278-0062}, + journal = {IEEE Transactions on Medical Imaging (TMI)}, + month = {jan}, + number = {1}, + pages = {106--121}, + pmid = {19116193}, + title = {{Highly Undersampled Magnetic Resonance Image Reconstruction via Homotopic L0-Minimization}}, + volume = {28}, + year = {2009} +} +@article{King2008, + author = {King, K}, + isbn = {1545-4428}, + journal = {Proceedings 16th Scientific Meeting, International Society for Magnetic Resonance in Medicine}, + pages = {1488}, + title = {{Combining Compressed Sensing and Parallel Imaging}}, + url = {/MyPathway2008/1488}, + volume = {Toronto}, + year = {2008} +} +@inproceedings{haarnoja2017reinforcement, + title={Reinforcement learning with deep energy-based policies}, + author={Haarnoja, Tuomas and Tang, Haoran and Abbeel, Pieter and Levine, Sergey}, + booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70}, + pages={1352--1361}, + year={2017}, + organization={JMLR. org} +} +@inproceedings{nachum2017bridging, + title={Bridging the gap between value and policy based reinforcement learning}, + author={Nachum, Ofir and Norouzi, Mohammad and Xu, Kelvin and Schuurmans, Dale}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2775--2785}, + year={2017} +} +@article{Pawar2015, + archivePrefix = {arXiv}, + arxivId = {arXiv:1407.5536v2}, + author = {Pawar, Kamlesh and Egan, Gary and Zhang, Jingxin}, + doi = {10.1371/journal.pone.0126386}, + editor = {Lenglet, Christophe}, + eprint = {arXiv:1407.5536v2}, + issn = {1932-6203}, + journal = {PLOS ONE}, + month = {may}, + number = {5}, + pages = {e0126386}, + title = {{Multichannel Compressive Sensing MRI Using Noiselet Encoding}}, + url = {http://dx.plos.org/10.1371/journal.pone.0126386}, + volume = {10}, + year = {2015} +} +@article{Cevher2010, + author = {Cevher, Volkan and Indyk, Piotr and Carin, Lawrence and Baraniuk, Richard}, + doi = {10.1109/MSP.2010.938029}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Cevher et al. - 2010 - Sparse Signal Recovery and Acquisition with Graphical Models.pdf:pdf}, + issn = {1053-5888}, + journal = {IEEE Signal Processing Magazine}, + month = {nov}, + number = {6}, + pages = {92--103}, + title = {{Sparse Signal Recovery and Acquisition with Graphical Models}}, + volume = {27}, + year = {2010} +} +@article{Jhooti2010, + title={A fully automatic and highly efficient navigator gating technique for high-resolution free-breathing acquisitions: Continuously adaptive windowing strategy}, + author={Jhooti, P and Keegan, J and Firmin, DN}, + journal={Magnetic resonance in medicine}, + volume={64}, + number={4}, + pages={1015--1026}, + year={2010}, + publisher={Wiley Online Library} +} +@inproceedings{Zhou2015, + author = {Zhou, Yihang and Shi, Chao and Ren, Fuquan and Lyu, Jingyuan and Liang, Dong and Ying, Leslie}, + booktitle = {Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI)}, + doi = {10.1109/ISBI.2015.7164015}, + isbn = {978-1-4799-2374-8}, + month = {apr}, + pages = {897--900}, + publisher = {IEEE}, + title = {{Accelerating MR parameter mapping using nonlinear manifold learning and supervised pre-imaging}}, + year = {2015} +} +@article{Adluru2011, + author = {Adluru, Ganesh and Chen, Liyong and Kim, Seong-Eun and Burgon, Nathan and Kholmovski, Eugene G and Marrouche, Nassir F and Dibella, Edward V R}, + doi = {10.1002/jmri.22808}, + issn = {1522-2586}, + journal = {Journal of Magnetic Resonance Imaging (JMRI)}, + month = {dec}, + number = {6}, + pages = {1465--71}, + pmid = {21972108}, + title = {{Three-dimensional late gadolinium enhancement imaging of the left atrium with a hybrid radial acquisition and compressed sensing.}}, + volume = {34}, + year = {2011} +} +@article{Tremoulheac2014, + author = {Tr{\'{e}}moulh{\'{e}}ac, Benjamin and Dikaios, Nikolaos and Atkinson, David and Arridge, Simon R}, + doi = {10.1109/TMI.2014.2321190}, + journal = {IEEE transactions on medical imaging (TMI)}, + number = {8}, + pages = {1689--1701}, + title = {{Dynamic MR Image Reconstruction – Separation From Undersample k-t Space via Low-Rank Plus Sparse Prior}}, + volume = {33}, + year = {2014} +} +@article{Haldar2011, + author = {Haldar, Justin P. and Hernando, Diego and Liang, Zhi Pei}, + doi = {10.1109/TMI.2010.2085084}, + isbn = {1558-0062 (Electronic)$\backslash$r0278-0062 (Linking)}, + issn = {02780062}, + journal = {IEEE Transactions on Medical Imaging (TMI)}, + keywords = {Compressed sensing,magnetic resonance imaging (MRI),radio-frequency encoding}, + number = {4}, + pages = {893--903}, + pmid = {20937579}, + title = {{Compressed-sensing MRI with random encoding}}, + volume = {30}, + year = {2011} +} +@book{McRobbie2006, + address = {Cambridge}, + author = {McRobbie, Donald W. and Moore, Elizabeth A. and Graves, Martin J. and Prince, Martin R.}, + doi = {10.1017/CBO9780511545405}, + isbn = {9780511545405}, + publisher = {Cambridge University Press}, + title = {{MRI from Picture to Proton}}, + year = {2006} +} +@article{Peng2015, + author = {Peng, Xi and Ying, Leslie and Liu, Qiegen and Zhu, Yanjie and Liu, Yuanyuan and Qu, Xiaobo and Liu, Xin and Zheng, Hairong and Liang, Dong}, + doi = {10.1002/mrm.25272}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Peng et al. - 2015 - Incorporating reference in parallel imaging and compressed sensing.pdf:pdf}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine}, + month = {apr}, + number = {4}, + pages = {1490--1504}, + pmid = {24771404}, + title = {{Incorporating reference in parallel imaging and compressed sensing}}, + volume = {73}, + year = {2015} +} +@article{Qu2010, + author = {Qu, X B and Zhang, W R and Guo, D and Cai, C B and Cai, S H and Chen, Z}, + doi = {Doi 10.1080/17415977.2010.492509\rPii 925482588}, + isbn = {1741-5977}, + issn = {1741-5977}, + journal = {Inverse Problems in Science and Engineering}, + number = {6}, + pages = {737--758}, + title = {{Iterative thresholding compressed sensing MRI based on contourlet transform}}, + volume = {18}, + year = {2010} +} +@inproceedings{Bhatia2015, + author = {Bhatia, Kanwal K. and Caballero, Jose and Price, Anthony N. and Sun, Ying and Hajnal, Jo V. and Rueckert, Daniel}, + booktitle = {Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI)}, + doi = {10.1007/978-3-319-24574-4_61}, + isbn = {9783319245737}, + issn = {16113349}, + pages = {510--518}, + pmid = {23285570}, + title = {{Fast Reconstruction of Accelerated Dynamic MRI Using Manifold Kernel Regression}}, + volume = {9351}, + year = {2015} +} +@article{Liu2014, + author = {Liu, Qiegen and Peng, Xi and Liu, Jianbo and Yang, Dingcheng and Liang, Dong}, + doi = {10.1155/2014/128596}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Liu et al. - 2014 - A Weighted Two-Level Bregman Method with Dictionary Updating for Nonconvex MR Image Reconstruction.pdf:pdf}, + issn = {1687-4188}, + journal = {International Journal of Biomedical Imaging (IJBI)}, + pages = {1--23}, + title = {{A Weighted Two-Level Bregman Method with Dictionary Updating for Nonconvex MR Image Reconstruction}}, + volume = {2014}, + year = {2014} +} +@article{Hong2011, + author = {Hong, Mingjian and Yu, Yeyang and Wang, Hua and Liu, Feng and Crozier, Stuart}, + doi = {10.1088/0031-9155/56/19/010}, + isbn = {0031-9155}, + issn = {1361-6560}, + journal = {Physics in Medicine and Biology}, + number = {19}, + pages = {6311--6325}, + pmid = {21896962}, + title = {{Compressed sensing MRI with singular value decomposition-based sparsity basis}}, + volume = {56}, + year = {2011} +} +@article{Cheng2015, + author = {Cheng, Joseph Y and Zhang, Tao and Ruangwattanapaisarn, Nichanan and Alley, Marcus T and Uecker, Martin and Pauly, John M and Lustig, Michael and Vasanawala, Shreyas S}, + doi = {10.1002/jmri.24785}, + issn = {1522-2586}, + journal = {Journal of magnetic resonance imaging (JMRI)}, + month = {aug}, + number = {2}, + pages = {407--20}, + pmid = {25329325}, + title = {{Free-breathing pediatric MRI with nonrigid motion correction and acceleration.}}, + volume = {42}, + year = {2015} +} +@article{Block2007, + author = {Block, Kai Tobias and Uecker, Martin and Frahm, Jens}, + doi = {10.1002/mrm.21236}, + journal = {Magnetic Resonance in Medicine}, + number = {6}, + pages = {1086--1098}, + title = {{Undersampled radial MRI with multiple coils. Iterative image reconstruction using a total variation constraint.}}, + volume = {57}, + year = {2007} +} +@article{Akcakaya2012, + author = {Ak{\c{c}}akaya, M and Rayatzadeh, H and Basha, Ta and Hong, Susie N and Chan, Raymond H and Kissinger, Kraig V and Hauser, Thomas H and Josephson, Mark E and Manning, Warren J}, + doi = {10.1148/radiol.12112489/-/DC1}, + journal = {Radiology}, + number = {3}, + pages = {691--699}, + title = {{Accelerated late gadolinium enhancement cardiac mr imaging with isotropic spatial resolution using compressed sensing: Initial experience}}, + volume = {264}, + year = {2012} +} +@article{Liang2009, + author = {Liang, Dong and Liu, Bo and Wang, Jiunjie and Ying, Leslie}, + doi = {10.1002/mrm.22161}, + file = {:privateUsers/gyang/Dropbox/Papers/CompressiveSensing/Liang09{\_}CSForSENSE.pdf:pdf}, + isbn = {0740-3194}, + issn = {1522-2594}, + journal = {Magnetic resonance in medicine}, + number = {6}, + pages = {1574--84}, + pmid = {19785017}, + title = {{Accelerating SENSE using compressed sensing.}}, + volume = {62}, + year = {2009} +} +@article{LI2013, + author = {LI, Qing and YANG, Xiao-mei and LI, Hong}, + doi = {10.3724/SP.J.1087.2012.00541}, + issn = {1001-9081}, + journal = {Journal of Computer Applications}, + number = {2}, + pages = {541--544}, + title = {{Compressed sensing-adaptive regularization for reconstruction of magnetic resonance image}}, + volume = {32}, + year = {2013} +} +@article{Zonoobi2014, + author = {Zonoobi, Dornoosh and Kassim, Ashraf a.}, + doi = {10.1016/j.media.2014.04.010}, + issn = {13618423}, + journal = {Medical Image Analysis}, + number = {6}, + pages = {857--865}, + pmid = {24874773}, + publisher = {Elsevier B.V.}, + title = {{A computationally efficient method for reconstructing sequences of MR images from undersampled k-space data}}, + volume = {18}, + year = {2014} +} +@article{QiegenLiu2013, + author = {Liu, Qiegen and Wang, Shanshan and Yang, Kun and Luo, Jianhua and Zhu, Yuemin and Liang, Dong}, + doi = {10.1109/TMI.2013.2256464}, + issn = {0278-0062}, + journal = {IEEE Transactions on Medical Imaging (TMI)}, + number = {7}, + pages = {1290--1301}, + title = {{Highly undersampled magnetic resonance image reconstruction using two-level Bregman method with dictionary updating}}, + volume = {32}, + year = {2013} +} +@article{Geethanath2012, + author = {Geethanath, Sairam and Baek, Hyeon-Man and Ganji, Sandeep K and Ding, Yao and Maher, Elizabeth a and Sims, Robert D and Choi, Changho and Lewis, Matthew a and Kodibagkar, Vikram D}, + doi = {10.1148/radiol.11111098}, + isbn = {1527-1315 (Electronic)$\backslash$r0033-8419 (Linking)}, + issn = {1527-1315}, + journal = {Radiology}, + number = {3}, + pages = {985--94}, + pmid = {22357898}, + title = {{Compressive sensing could accelerate 1H MR metabolic imaging in the clinic.}}, + volume = {262}, + year = {2012} +} +@article{Baraniuk2007, + author = {Baraniuk, Richard}, + doi = {10.1109/MSP.2007.4286571}, + isbn = {1053-5888 VO - 24}, + issn = {1053-5888}, + journal = {IEEE Signal Processing Magazine}, + month = {jul}, + number = {4}, + pages = {118--121}, + pmid = {19158952}, + title = {{Compressive Sensing}}, + volume = {24}, + year = {2007} +} +@article{Otazo2015, + author = {Otazo, Ricardo and Cand{\`{e}}s, Emmanuel and Sodickson, Daniel K.}, + doi = {10.1002/mrm.25240}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine}, + month = {mar}, + number = {3}, + pages = {1125--1136}, + title = {{Low-rank plus sparse matrix decomposition for accelerated dynamic MRI with separation of background and dynamic components}}, + volume = {73}, + year = {2015} +} +@inproceedings{Chartrand2009, + author = {Chartrand, Rick}, + doi = {10.1109/ISBI.2009.5193034}, + isbn = {9781424439324}, + issn = {1945-7928}, + booktitle = {Proceedings of the IEEE International Symposium on Biomedical Imaging (ISBI)}, + number = {1}, + pages = {262--265}, + title = {{Fast algorithms for nonconvex compressive sensing: MRI reconstruction from very few data}}, + year = {2009} +} +@article{Wu2013, + title={Accelerated MR diffusion tensor imaging using distributed compressed sensing}, + author={Wu, Yin and Zhu, Yan-Jie and Tang, Qiu-Yang and Zou, Chao and Liu, Wei and Dai, Rui-Bin and Liu, Xin and Wu, Ed X and Ying, Leslie and Liang, Dong}, + journal={Magnetic resonance in medicine}, + volume={71}, + number={2}, + pages={763--772}, + year={2014}, + publisher={Wiley Online Library} +} + +@article{Murphy2012, + author = {Murphy, Mark and Alley, Marcus and Demmel, James and Keutzer, Kurt and Vasanawala, Shreyas and Lustig, Michael}, + doi = {10.1109/TMI.2012.2188039}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Murphy et al. - 2012 - Fast l₁-SPIRiT compressed sensing parallel imaging MRI scalable parallel implementation and clinically feasible.pdf:pdf}, + isbn = {0278-0062}, + issn = {1558-254X}, + journal = {IEEE Transactions on Medical Imaging (TMI)}, + number = {6}, + pages = {1250--62}, + pmid = {22345529}, + title = {{Fast l₁-SPIRiT compressed sensing parallel imaging MRI: scalable parallel implementation and clinically feasible runtime.}}, + volume = {31}, + year = {2012} +} +@article{Khare2012, + author = {Khare, Kedar and Hardy, Christopher J and King, Kevin F and Turski, Patrick A and Marinelli, Luca}, + doi = {10.1002/mrm.24143}, + isbn = {0740-3194}, + issn = {1522-2594}, + journal = {Magnetic Resonance in Medicine }, + pages = {1450--1457}, + pmid = {22266597}, + title = {{Accelerated MR imaging using compressive sensing with no free parameters}}, + volume = {68}, + year = {2012} +} +@article{Uecker2014, + author = {Uecker, Martin and Lai, Peng and Murphy, Mark J. and Virtue, Patrick and Elad, Michael and Pauly, John M. and Vasanawala, Shreyas S. and Lustig, Michael}, + doi = {10.1002/mrm.24751}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine }, + number = {3}, + pages = {990--1001}, + title = {{ESPIRiT-an eigenvalue approach to autocalibrating parallel MRI: Where SENSE meets GRAPPA}}, + volume = {71}, + year = {2014} +} +@article{Candes2008, + archivePrefix = {arXiv}, + arxivId = {arXiv:1307.1360v1}, + author = {Candes, E.J. and Wakin, M.B.}, + doi = {10.1109/MSP.2007.914731}, + eprint = {arXiv:1307.1360v1}, + isbn = {1053-5888 VO - 25}, + issn = {1053-5888}, + journal = {IEEE Signal Processing Magazine}, + number = {2}, + pages = {21--30}, + pmid = {4472240}, + title = {{An Introduction To Compressive Sampling}}, + volume = {25}, + year = {2008} +} +@article{Miao2013, + author = {Miao, Jun and Guo, Weihong and Narayan, Sreenath and Wilson, David L}, + doi = {10.1016/j.mri.2012.06.028}, + file = {:privateUsers/gyang/Library/Application Support/Mendeley Desktop/Downloaded/Miao et al. - 2013 - A simple application of compressed sensing to further accelerate partially parallel imaging.pdf:pdf}, + isbn = {0730-725X}, + issn = {1873-5894}, + journal = {Journal of Magnetic Resonance Imaging (JMRI)}, + month = {jan}, + number = {1}, + pages = {75--85}, + pmid = {22902065}, + publisher = {Elsevier Inc.}, + title = {{A simple application of compressed sensing to further accelerate partially parallel imaging.}}, + volume = {31}, + year = {2013} +} +@article{Oh2014, + author = {Oh, Heeseok and Lee, Sanghoon}, + doi = {10.1016/j.mri.2012.11.008}, + isbn = {0730-725X}, + issn = {0730725X}, + journal = {Journal of Magnetic Resonance Imaging (JMRI)}, + number = {3}, + pages = {270--280}, + pmid = {24332887}, + publisher = {Elsevier Inc.}, + title = {{Visually weighted reconstruction of compressive sensing MRI}}, + url = {http://dx.doi.org/10.1016/j.mri.2012.11.008}, + volume = {32}, + year = {2014} +} +@article{Chang2010, + author = {Chang, Ching-hua and Ji, Jim}, + doi = {10.1002/mrm.22481}, + isbn = {0740-3194}, + issn = {1522-2594}, + journal = {Magnetic Resonance in Medicine}, + pages = {1135--1139}, + pmid = {20564584}, + title = {{Compressed Sensing MRI with Multichannel Data Using Multicore Processors}}, + volume = {1139}, + year = {2010} +} +@article{Majumdar2012, + author = {Majumdar, Angshul and Ward, Rabab K.}, + doi = {10.1016/j.image.2012.08.002}, + isbn = {0923-5965}, + issn = {09235965}, + journal = {Signal Processing: Image Communication}, + keywords = {Compressed Sensing,MRI,Non-convex algorithms}, + number = {9}, + pages = {1035--1048}, + publisher = {Elsevier}, + title = {{On the choice of Compressed Sensing priors and sparsifying transforms for MR image reconstruction: An experimental study}}, + volume = {27}, + year = {2012} +} +@article{Donoho2006, + author = {Donoho, David L.}, + doi = {10.1109/TIT.2006.871582}, + issn = {0018-9448}, + journal = {IEEE Transactions on Information Theory}, + number = {4}, + pages = {1289--1306}, + title = {{Compressed sensing}}, + volume = {52}, + year = {2006} +} +@article{Huang2011, + author = {Huang, Junzhou and Zhang, Shaoting and Metaxas, Dimitris}, + doi = {10.1016/j.media.2011.06.001}, + issn = {13618415}, + journal = {Medical Image Analysis}, + number = {5}, + pages = {670--679}, + publisher = {Elsevier B.V.}, + title = {{Efficient MR image reconstruction for compressed MR imaging}}, + volume = {15}, + year = {2011} +} +@article{Akcakaya2012a, + archivePrefix = {arXiv}, + arxivId = {NIHMS150003}, + author = {Ak{\c{c}}akaya, Mehmet and Basha, Tamer A and Chan, Raymond H and Rayatzadeh, Hussein and Kissinger, Kraig V and Goddu, Beth and Goepfert, Lois A and Manning, Warren J and Nezafat, Reza}, + doi = {10.1002/mrm.24242}, + eprint = {NIHMS150003}, + isbn = {2122633255}, + issn = {1522-2594}, + journal = {Magnetic resonance in medicine}, + number = {5}, + pages = {1434--43}, + pmid = {22392654}, + title = {{Accelerated contrast-enhanced whole-heart coronary MRI using low-dimensional-structure self-learning and thresholding.}}, + volume = {67}, + year = {2012} +} +@book{Kwong2008, + editor = {Kwong, Raymond Y.}, + publisher = {Humana Press}, + title = {{Cardiovascular Magnetic Resonance Imaging}}, + year = {2008} +} +@article{Weingartner2015a, + author = {Weing{\"{a}}rtner, Sebastian and Ak{\c{c}}akaya, Mehmet and Roujol, S{\'{e}}bastien and Basha, Tamer and Tschabrunn, Cory and Berg, Sophie and Anter, Elad and Nezafat, Reza}, + doi = {10.1002/mrm.25495}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine}, + number = {4}, + pages = {1032--1041}, + title = {{Free-breathing combined three-dimensional phase sensitive late gadolinium enhancement and T {\textless}sub{\textgreater}1{\textless}/sub{\textgreater} mapping for myocardial tissue characterization}}, + volume = {74}, + year = {2015} +} +@article{Hollingsworth2014, + author = {Hollingsworth, Kieren G. and Higgins, David M. and McCallum, Michelle and Ward, Louise and Coombs, Anna and Straub, Volker}, + doi = {10.1002/mrm.25072}, + issn = {1522-2594}, + journal = {Magnetic resonance in medicine}, + month = {dec}, + number = {6}, + pages = {1610--9}, + pmid = {24347306}, + title = {{Investigating the quantitative fidelity of prospectively undersampled chemical shift imaging in muscular dystrophy with compressed sensing and parallel imaging reconstruction.}}, + volume = {72}, + year = {2014} +} +@article{Vincenti2014, + author = {Vincenti, Gabriella and Monney, Pierre and Chaptinel, J{\'{e}}r{\^{o}}me and Rutz, Tobias and Coppo, Simone and Zenge, Michael O and Schmidt, Michaela and Nadar, Mariappan S and Piccini, Davide and Ch{\`{e}}vre, Pascal and Stuber, Matthias and Schwitter, Juerg}, + doi = {10.1016/j.jcmg.2014.04.016}, + isbn = {1936-878X}, + issn = {1876-7591}, + journal = {JACC. Cardiovascular imaging}, + month = {sep}, + number = {9}, + pages = {882--92}, + pmid = {25129517}, + title = {{Compressed sensing single-breath-hold CMR for fast quantification of LV function, volumes, and mass.}}, + volume = {7}, + year = {2014} +} + +@article{Mann2015, + author = {Mann, Louis W. and Higgins, David M. and Peters, Carl N. and Cassidy, Sophie and Hodson, Kenneth K. and Coombs, Anna and Taylor, Roy and Hollingsworth, Kieren G.}, + doi = {10.1148/radiol.2015150320}, + issn = {1527-1315}, + journal = {Radiology}, + month = {jul}, + number = {0}, + pages = {150320}, + pmid = {26218662}, + title = {{Accelerating MR Imaging Liver Steatosis Measurement Using Combined Compressed Sensing and Parallel Imaging: A Quantitative Evaluation.}}, + volume = {000}, + year = {2015} +} +@article{Qu2012, + author = {Qu, Xiaobo and Guo, Di and Ning, Bende and Hou, Yingkun and Lin, Yulan and Cai, Shuhui and Chen, Zhong}, + doi = {10.1016/j.mri.2012.02.019}, + issn = {1873-5894}, + journal = {Magnetic Resonance Imaging}, + number = {7}, + pages = {964--977}, + pmid = {22504040}, + title = {{Undersampled MRI reconstruction with patch-based directional wavelets}}, + volume = {30}, + year = {2012} +} +@article{Peng2016, + author = {Peng, Zhanglin and Li, Ya and Cai, Zhaoquan and Lin, Liang}, + doi = {10.1016/j.neucom.2015.07.116}, + issn = {09252312}, + journal = {Neurocomputing}, + month = {feb}, + pages = {36--45}, + title = {{Deep Boosting: Joint feature selection and analysis dictionary learning in hierarchy}}, + volume = {178}, + year = {2016} +} +@inproceedings{Chakraborty2008, + author = {Chakraborty, Amit}, + doi = {10.1109/CVPR.2008.4587391}, + isbn = {978-1-4244-2242-5}, + issn = {1063-6919}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + pages = {1--8}, + title = {{An efficient algorithm for compressed MR imaging using total variation and wavelets}}, + year = {2008} +} +@article{Ajraoui2013, + author = {Ajraoui, Salma and Parra-Robles, J and Wild, J M}, + doi = {10.1002/mrm.24252}, + file = {:privateUsers/gyang/Dropbox/Papers/CompressiveSensing/Ajraoui13{\_}QuantiativeAnalysisCSGasImaging.pdf:pdf}, + isbn = {0740-3194}, + issn = {1522-2594}, + journal = {Magnetic resonance in medicine}, + month = {feb}, + number = {2}, + pages = {360--9}, + pmid = {22473679}, + title = {{Incorporation of prior knowledge in compressed sensing for faster acquisition of hyperpolarized gas images.}}, + volume = {69}, + year = {2013} +} +@article{Weingartner2015, + author = {Weing{\"{a}}rtner, Sebastian and Ak{\c{c}}akaya, Mehmet and Roujol, S{\'{e}}bastien and Basha, Tamer and Stehning, Christian and Kissinger, Kraig V. and Goddu, Beth and Berg, Sophie and Manning, Warren J. and Nezafat, Reza}, + doi = {10.1002/mrm.25124}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine}, + keywords = {diffused fibrosis,myocardial t 1 mapping,naviga-,quantitative cardiac mri,tor gating}, + number = {1}, + pages = {214--222}, + title = {{Free-breathing post-contrast three-dimensional T {\textless}sub{\textgreater}1{\textless}/sub{\textgreater} mapping: Volumetric assessment of myocardial T {\textless}sub{\textgreater}1{\textless}/sub{\textgreater} values}}, + volume = {73}, + year = {2015} +} +@article{Liu2009, + author = {Liu, Bo and King, Kevin and Steckner, Michael and Xie, Jun and Sheng, Jinhua and Ying, Leslie}, + doi = {10.1002/mrm.21799}, + isbn = {0740-3194}, + issn = {1522-2594}, + journal = {Magnetic resonance in medicine}, + keywords = {Bregman iteration,Compressed sensing,Parallel imaging,SENSE,Total variation regularization}, + month = {jan}, + number = {1}, + pages = {145--52}, + pmid = {19097223}, + title = {{Regularized sensitivity encoding (SENSE) reconstruction using Bregman iterations.}}, + volume = {61}, + year = {2009} +} +@article{Chaari2011, + author = {Cha{\^{a}}ri, Lotfi and Pesquet, Jean-Christophe and Benazza-Benyahia, Amel and Ciuciu, Philippe}, + doi = {10.1016/j.media.2010.08.001}, + issn = {13618415}, + journal = {Medical Image Analysis}, + month = {apr}, + number = {2}, + pages = {185--201}, + title = {{A wavelet-based regularized reconstruction algorithm for SENSE parallel MRI with applications to neuroimaging}}, + volume = {15}, + year = {2011} +} +@article{Loughran2015, + author = {Loughran, Thomas and Higgins, David M and McCallum, Michelle and Coombs, Anna and Straub, Volker and Hollingsworth, Kieren G}, + doi = {10.1148/radiol.14141191}, + issn = {1527-1315}, + journal = {Radiology}, + month = {may}, + number = {2}, + pages = {570--8}, + pmid = {25575118}, + title = {{Improving highly accelerated fat fraction measurements for clinical trials in muscular dystrophy: origin and quantitative effect of R2* changes.}}, + volume = {275}, + year = {2015} +} +@article{Liu2015, + author = {Liu, Yiguang and Yang, Songfan and Wu, Pengfei and Li, Chunguang and Yang, Menglong}, + doi = {10.1109/TNNLS.2015.2496964}, + issn = {2162-2388}, + journal = {IEEE transactions on neural networks and learning systems (TNNLS)}, + month = {nov}, + number = {1}, + pages = {1--11}, + pmid = {26595933}, + title = {{L₁-Norm Low-Rank Matrix Decomposition by Neural Networks and Mollifiers.}}, + year = {2015} +} +@article{Zhang2015, + author = {Zhang, Tao and Cheng, Joseph Y. and Potnick, Aaron G. and Barth, Richard A. and Alley, Marcus T. and Uecker, Martin and Lustig, Michael and Pauly, John M. and Vasanawala, Shreyas S.}, + doi = {10.1002/jmri.24551}, + isbn = {1522-2586 (Electronic)$\backslash$r1053-1807 (Linking)}, + issn = {15222586}, + journal = {Journal of Magnetic Resonance Imaging (JMRI)}, + number = {2}, + pages = {460--473}, + pmid = {24375859}, + title = {{Fast pediatric 3D free-breathing abdominal dynamic contrast enhanced MRI with high spatiotemporal resolution}}, + volume = {41}, + year = {2015} +} +@article{Song2014, + author = {Song, Ying and Zhu, Zhen and Lu, Yang and Liu, Qiegen and Zhao, Jun}, + doi = {10.1002/mrm.24734}, + isbn = {9781424479290}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine}, + number = {3}, + pages = {1285--1298}, + pmid = {23554046}, + title = {{Reconstruction of magnetic resonance imaging by three-dimensional dual-dictionary learning}}, + volume = {71}, + year = {2014} +} +@article{Yanhua2014, + author = {Yanhua, Wang and Ying, L}, + doi = {10.1109/TBME.2013.2294939}, + isbn = {0018-9294}, + issn = {1558-2531}, + journal = {IEEE Transactions on Biomedical Engineering}, + number = {4}, + pages = {1109--1120}, + pmid = {24658236}, + title = {{Compressed Sensing Dynamic Cardiac Cine MRI Using Learned Spatiotemporal Dictionary}}, + volume = {61}, + year = {2014} +} +@article{Lingala2013, + archivePrefix = {arXiv}, + arxivId = {NIHMS150003}, + author = {Lingala, Sajan Goud and Jacob, Mathews}, + doi = {10.1109/TMI.2013.2255133}, + eprint = {NIHMS150003}, + isbn = {9781457718588}, + issn = {1558-254X}, + journal = {IEEE Transactions on Medical Imaging (TMI)}, + number = {6}, + pages = {1132--1145}, + pmid = {23542951}, + title = {{Blind compressive sensing dynamic MRI.}}, + volume = {32}, + year = {2013} +} +@article{Caballero2014, + author = {Caballero, Jose and Price, Anthony N and Rueckert, Daniel and Hajnal, Joseph V}, + doi = {10.1109/TMI.2014.2301271}, + isbn = {0278-0062 VO - 33}, + issn = {1558-254X}, + journal = {IEEE Transactions on Medical Imaging (TMI)}, + number = {4}, + pages = {979--994}, + pmid = {24710166}, + title = {{Dictionary learning and time sparsity for dynamic MR data reconstruction}}, + volume = {33}, + year = {2014} +} +@article{Huang2014, + arxivId = {arXiv:1302.2712v2}, + author = {Huang, Yue and Paisley, J and Lin, Qin and Ding, Xinghao and Fu, Xueyang and Zhang, Xiaoping}, + doi = {10.1109/TIP.2014.2360122}, + eprint = {arXiv:1302.2712v2}, + isbn = {1057-7149 VO - 23}, + issn = {10577149}, + journal = {IEEE Transaction on Image Processing (TIP)}, + number = {12}, + pages = {5007--5019}, + pmid = {25265609}, + title = {{Bayesian nonparametric dictionary learning for compressed sensing MRI}}, + volume = {23}, + year = {2014} +} +@article{Tosic2011, + author = {Tosic, Ivana; and Frossard, Pascal}, + doi = {10.1109/MSP.2010.939537}, + journal = {IEEE Signal Processing Magazine}, + number = {2}, + pages = {27--38}, + title = {{Dictionary Learning, What is the right representation for my signal?}}, + volume = {28}, + year = {2011} +} +@article{Lai2016, + author = {Lai, Zongying and Qu, Xiaobo and Liu, Yunsong and Guo, Di and Ye, Jing and Zhan, Zhifang and Chen, Zhong}, + doi = {10.1016/j.media.2015.05.012}, + issn = {13618415}, + journal = {Medical Image Analysis}, + pages = {93--104}, + publisher = {Elsevier Ltd.}, + title = {{Image reconstruction of compressed sensing MRI using graph-based redundant wavelet transform}}, + volume = {27}, + year = {2016} +} +@article{Akcakaya2011, + author = {Ak{\c{c}}akaya, Mehmet and Basha, Tamer a. and Goddu, Beth and Goepfert, Lois a. and Kissinger, Kraig V. and Tarokh, Vahid and Manning, Warren J. and Nezafat, Reza}, + doi = {10.1002/mrm.22841}, + isbn = {1522-2594 (Electronic)$\backslash$r0740-3194 (Linking)}, + issn = {07403194}, + journal = {Magnetic Resonance in Medicine}, + number = {3}, + pages = {756--767}, + pmid = {21465542}, + title = {{Low-dimensional-structure self-learning and thresholding: Regularization beyond compressed sensing for MRI Reconstruction}}, + volume = {66}, + year = {2011} +} +@article{Hansen2013, + author = {Hansen, Michael Schacht and S{\o}rensen, Thomas Sangild}, + doi = {10.1002/mrm.24389}, + issn = {1522-2594}, + journal = {Magnetic resonance in medicine}, + month = {jun}, + number = {6}, + pages = {1768--76}, + pmid = {22791598}, + title = {{Gadgetron: an open source framework for medical image reconstruction.}}, + volume = {69}, + year = {2013} +} +@article{Hollingsworth2015, + author = {Hollingsworth, Kieren Grant}, + doi = {10.1088/0031-9155/60/21/R297}, + issn = {0031-9155}, + journal = {Physics in Medicine and Biology}, + number = {21}, + pages = {297--322}, + publisher = {IOP Publishing}, + title = {{Reducing acquisition time in clinical MRI by data undersampling and compressed sensing reconstruction}}, + volume = {60}, + year = {2015} +} +@article{Lustig2008, + author = {Lustig, Michael and Donoho, D.L. and Santos, J.MIEEE Signal Process. Mag.. and Pauly, J.M.}, + doi = {10.1109/MSP.2007.914728}, + eprint = {1204.4227v1}, + isbn = {0018-9448}, + issn = {1053-5888}, + journal = {IEEE Signal Processing Magazine}, + number = {2}, + pages = {72--82}, + pmid = {1614066}, + title = {{Compressed sensing MRI}}, + volume = {25}, + year = {2008} +} + +@article{crum2004non, + title={Non-rigid image registration: theory and practice}, + author={Crum, William R and Hartkens, Thomas and Hill, DLG}, + journal={The British journal of radiology}, + volume={77}, + number={suppl\_2}, + pages={S140--S153}, + year={2004}, + publisher={British Institute of Radiology} +} + + +@article{rajpurkar2017chexnet, + title={Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning}, + author={Rajpurkar, Pranav and Irvin, Jeremy and Zhu, Kaylie and Yang, Brandon and Mehta, Hershel and Duan, Tony and Ding, Daisy and Bagul, Aarti and Langlotz, Curtis and Shpanskaya, Katie and Matthew P. Lungren and Andrew Y. Ng}, + journal={arXiv preprint arXiv:1711.05225}, + year={2017} +} +%% brain tumor +@article{mazzara2004brain, + title={Brain tumor target volume determination for radiation treatment planning through automated MRI segmentation}, + author={Mazzara, Gloria P and Velthuizen, Robert P and Pearlman, James L and Greenberg, Harvey M and Wagner, Henry}, + journal={International Journal of Radiation Oncology* Biology* Physics}, + volume={59}, + number={1}, + pages={300--312}, + year={2004}, + publisher={Elsevier} +} +@article{yamahara2010morphological, + title={Morphological and flow cytometric analysis of cell infiltration in glioblastoma: a comparison of autopsy brain and neuroimaging}, + author={Yamahara, Takahiro and Numa, Yoshihiro and Oishi, Tetsuya and Kawaguchi, Takuya and Seno, Toshitaka and Asai, Akio and Kawamoto, Keiji}, + journal={Brain tumor pathology}, + volume={27}, + number={2}, + pages={81--87}, + year={2010}, + publisher={Springer} +} +@article{bauer2013survey, + title={A survey of MRI-based medical image analysis for brain tumor studies}, + author={Bauer, Stefan and Wiest, Roland and Nolte, Lutz-P and Reyes, Mauricio}, + journal={Physics in Medicine \& Biology}, + volume={58}, + number={13}, + pages={R97}, + year={2013}, + publisher={IOP Publishing} +} +@article{furnari2007malignant, + title={Malignant astrocytic glioma: genetics, biology, and paths to treatment}, + author={Furnari, Frank B and Fenton, Tim and Bachoo, Robert M and Mukasa, Akitake and Stommel, Jayne M and Stegh, Alexander and Hahn, William C and Ligon, Keith L and Louis, David N and Brennan, Cameron and others}, + journal={Genes \& development}, + volume={21}, + number={21}, + pages={2683--2710}, + year={2007}, + publisher={Cold Spring Harbor Lab} +} +@article{yang2014discrimination, + title={Discrimination between glioblastoma multiforme and solitary metastasis using morphological features derived from the p: q tensor decomposition of diffusion tensor imaging}, + author={Yang, Guang and Jones, Timothy L and Barrick, Thomas R and Howe, Franklyn A}, + journal={NMR in Biomedicine}, + volume={27}, + number={9}, + pages={1103--1111}, + year={2014}, + publisher={Wiley Online Library} +} +%% MRI brain tumour segmentation UNsupervised +@inproceedings{szilagyi2015automatic, + title={Automatic brain tumor segmentation in multispectral MRI volumes using a fuzzy c-means cascade algorithm}, + author={Szil{\'a}gyi, L{\'a}szl{\'o} and Lefkovits, L{\'a}szl{\'o} and Benyo, Balazs}, + booktitle={The 12th International Conference on Fuzzy Systems and Knowledge Discovery (FSKD)}, + pages={285--291}, + year={2015}, + organization={IEEE} +} +@article{mei2015analysis, + title={Analysis of neoplastic lesions in magnetic resonance imaging using self-organizing maps}, + author={Mei, Paulo Afonso and de Carvalho Carneiro, Cleyton and Fraser, Stephen J and Min, Li Li and Reis, Fabiano}, + journal={Journal of the neurological sciences}, + volume={359}, + number={1-2}, + pages={78--83}, + year={2015}, + publisher={Elsevier} +} +@article{juan2015automated, + title={Automated glioblastoma segmentation based on a multiparametric structured unsupervised classification}, + author={Juan-Albarrac{\'\i}n, Javier and Fuster-Garcia, Elies and Manj{\'o}n, Jos{\'e} V and Robles, Montserrat and Aparici, F and Mart{\'\i}-Bonmat{\'\i}, L and Garc{\'\i}a-G{\'o}mez, Juan M}, + journal={PLoS One}, + volume={10}, + number={5}, + pages={e0125143}, + year={2015}, + publisher={Public Library of Science} +} +@article{rajendran2012fuzzy, + title={Fuzzy clustering and deformable model for tumor segmentation on MRI brain image: a combined approach}, + author={Rajendran, A and Dhanasekaran, R}, + journal={Procedia Engineering}, + volume={30}, + pages={327--333}, + year={2012}, + publisher={Elsevier} +} +@article{hsieh2011automatic, + title={Automatic segmentation of meningioma from non-contrasted brain MRI integrating fuzzy clustering and region growing}, + author={Hsieh, Thomas M and Liu, Yi-Min and Liao, Chun-Chih and Xiao, Furen and Chiang, I-Jen and Wong, Jau-Min}, + journal={BMC medical informatics and decision making}, + volume={11}, + number={1}, + pages={54}, + year={2011}, + publisher={BioMed Central} +} +%% MRI brain tumour segmentation Supervised +@article{wu2014brain, + title={Brain tumor detection and segmentation in a CRF (conditional random fields) framework with pixel-pairwise affinity and superpixel-level features}, + author={Wu, Wei and Chen, Albert YC and Zhao, Liang and Corso, Jason J}, + journal={International Journal of Computer Assisted Radiology and Surgery}, + volume={9}, + number={2}, + pages={241--253}, + year={2014}, + publisher={Springer} +} +@inproceedings{pinto2015brain, + title={Brain tumour segmentation based on extremely randomized forest with high-level features}, + author={Pinto, Adriano and Pereira, S{\'e}rgio and Correia, Higino and Oliveira, Jorge and Rasteiro, Deolinda MLD and Silva, Carlos A}, + booktitle={The 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)}, + pages={3037--3040}, + year={2015}, + organization={IEEE} +} +@article{goetz2014extremely, + title={Extremely randomized trees based brain tumor segmentation}, + author={Goetz, Michael and Weber, Christian and Bloecher, Josiah and Stieltjes, Bram and Meinzer, Hans-Peter and Maier-Hein, Klaus}, + journal={Proceeding of BRATS challenge-MICCAI}, + pages={006--011}, + year={2014} +} +@article{soltaninejad2017automated, + title={Automated brain tumour detection and segmentation using superpixel-based extremely randomized trees in FLAIR MRI}, + author={Soltaninejad, Mohammadreza and Yang, Guang and Lambrou, Tryphon and Allinson, Nigel and Jones, Timothy L and Barrick, Thomas R and Howe, Franklyn A and Ye, Xujiong}, + journal={International Journal of Computer Assisted Radiology and Surgery}, + volume={12}, + number={2}, + pages={183--203}, + year={2017}, + publisher={Springer} +} +@article{jafari2011automatic, + title={Automatic brain tissue detection in MRI images using seeded region growing segmentation and neural network classification}, + author={Jafari, Mehdi and Kasaei, Shohreh}, + journal={Australian Journal of Basic and Applied Sciences}, + volume={5}, + number={8}, + pages={1066--1079}, + year={2011} +} +@inproceedings{subbanna2014iterative, + title={Iterative multilevel MRF leveraging context and voxel information for brain tumour segmentation in MRI}, + author={Subbanna, Nagesh and Precup, Doina and Arbel, Tal}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + pages={400--405}, + year={2014} +} +%% MRI CNN +@article{pereira2016brain, + title={Brain tumor segmentation using convolutional neural networks in MRI images}, + author={Pereira, S{\'e}rgio and Pinto, Adriano and Alves, Victor and Silva, Carlos A}, + journal={IEEE Transactions on Medical Imaging (TMI)}, + volume={35}, + number={5}, + pages={1240--1251}, + year={2016}, + publisher={IEEE} +} +@article{havaei2017brain, + title={Brain tumor segmentation with deep neural networks}, + author={Havaei, Mohammad and Davy, Axel and Warde-Farley, David and Biard, Antoine and Courville, Aaron and Bengio, Yoshua and Pal, Chris and Jodoin, Pierre-Marc and Larochelle, Hugo}, + journal={Medical image analysis}, + volume={35}, + pages={18--31}, + year={2017}, + publisher={Elsevier} +} +@article{kamnitsas2017efficient, + title={Efficient multi-scale 3D CNN with fully connected CRF for accurate brain lesion segmentation}, + author={Kamnitsas, Konstantinos and Ledig, Christian and Newcombe, Virginia FJ and Simpson, Joanna P and Kane, Andrew D and Menon, David K and Rueckert, Daniel and Glocker, Ben}, + journal={Medical image analysis}, + volume={36}, + pages={61--78}, + year={2017}, + publisher={Elsevier} +} +@incollection{drozdzal2016importance, + title={The importance of skip connections in biomedical image segmentation}, + author={Drozdzal, Michal and Vorontsov, Eugene and Chartrand, Gabriel and Kadoury, Samuel and Pal, Chris}, + booktitle={Deep Learning and Data Labeling for Medical Applications}, + pages={179--187}, + year={2016}, + publisher={Springer} +} +%% elastic transformation +@inproceedings{simard2003best, + title={Best practices for convolutional neural networks applied to visual document analysis}, + author={Simard, Patrice Y and Steinkraus, Dave and Platt, John C}, + booktitle = {Proceedings of the International Conference on Document Analysis and Recognition (ICDAR)}, + pages={958}, + year={2003}, + organization={IEEE} +} +%% MRI dataset +@article{menze2015multimodal, + title={The multimodal brain tumor image segmentation benchmark (BRATS)}, + author={Menze, Bjoern H and Jakab, Andras and Bauer, Stefan and Kalpathy-Cramer, Jayashree and Farahani, Keyvan and Kirby, Justin and Burren, Yuliya and Porz, Nicole and Slotboom, Johannes and Wiest, Roland and Levente Lanczi and Elizabeth Gerstner and Marc-André Weber and Tal Arbel and Brian B. Avants and Nicholas Ayache and Patricia Buendia and D. Louis Collins and Nicolas Cordier and Jason J. Corso and Antonio Criminisi and Tilak Das and Hervé Delingette and Çağatay Demiralp and Christopher R. Durst and Michel Dojat and Senan Doyle and Joana Festa and Florence Forbes and Ezequiel Geremia}, + journal={IEEE transactions on medical imaging}, + volume={34}, + number={10}, + pages={1993}, + year={2015}, + publisher={NIH Public Access} +} + +%% privacy +@article{YuanPrivacyBackPropa2014, + title={Privacy preserving back-propagation neural network learning made practical with cloud computing}, + author={Yuan, Jiawei and Yu, Shucheng}, + journal={IEEE Transactions on Parallel and Distributed Systems}, + volume={25}, + number={1}, + pages={212--221}, + year={2014}, + publisher={IEEE} +} +@article{BondeReviewPrivacyBackPropa2014, + title={Review techniques of data privacy in cloud using back propagation neural network}, + author={Bonde, DJ and Akib, Shaikh and Shubhangi, Pokharkar and Surbhi, Auti and Satish, Shelke}, + journal={International Journal of Emerging Technology and Advanced Engineering}, + volume={4}, + number={2}, + pages={15}, + year={2014} +} +@article{BansalPrivacyBackPropa2010, + title={Privacy preserving back-propagation neural network learning over arbitrarily partitioned data}, + author={Bansal, Ankur and Chen, Tingting and Zhong, Sheng}, + journal={Neural Computing and Applications}, + volume={20}, + number={1}, + pages={143--150}, + year={2011}, + publisher={Springer} +} +% privacy for hyperplane decision, Naïve Bayes, and decision trees +@inproceedings{BostLearningEncryptedData, + title={Machine learning classification over encrypted data}, + author={Bost, Raphael and Popa, Raluca Ada and Tu, Stephen and Goldwasser, Shafi}, + booktitle={The Network and Distributed System Security Symposium (NDSS)}, + year={2015} +} +% +@inproceedings{GraepelLearningEncryptedData, + title={ML confidential: Machine learning on encrypted data}, + author={Graepel, Thore and Lauter, Kristin and Naehrig, Michael}, + booktitle={International Conference on Information Security and Cryptology (ICISC)}, + pages={1--21}, + year={2012}, + organization={Springer} +} +@article{garcia2015edge, + title={Edge-centric computing: Vision and challenges}, + author={Garcia Lopez, Pedro and Montresor, Alberto and Epema, Dick and Datta, Anwitaman and Higashino, Teruo and Iamnitchi, Adriana and Barcellos, Marinho and Felber, Pascal and Riviere, Etienne}, + journal={ACM SIGCOMM Computer Communication Review}, + volume={45}, + number={5}, + pages={37--42}, + year={2015}, + publisher={ACM} +} +%% EEG paper +%10/20 +@misc{10/20System, + author = {{Trans Cranial Technologies Ltd}}, + title = {{10 / 20 System Positioning Manual}}, + howpublished = "\url{www.trans-cranial.com}", + year = {2012}, +} +@book{niedermeyer2005electroencephalography, + title={Electroencephalography: basic principles, clinical applications, and related fields}, + author={Niedermeyer, Ernst and da Silva, FH Lopes}, + year={2005}, + publisher={Lippincott Williams \& Wilkins} +} +% Sleep disease +@article{Ohayon_Insomnia2002, + author = {Ohayon, Maurice M.}, + doi = {10.1053/smrv.2002.0186}, + file = {:Users/haodong/Documents/Papers/2002/Ohayon - 2002 - Epidemiology of insomnia What we know and what we still need to learn.pdf:pdf}, + isbn = {1087-0792 (Print)$\backslash$r1087-0792 (Linking)}, + issn = {10870792}, + journal = {Sleep Medicine Reviews}, + keywords = {DSM-IV,Epidemiology,Insomnia,Mental disorders}, + mendeley-groups = {EEG/sleep disease}, + number = {2}, + pages = {97--111}, + pmid = {12531146}, + title = {{Epidemiology of insomnia: What we know and what we still need to learn}}, + volume = {6}, + year = {2002} +} +% Sleep standard +@book{Rechtschaffen1968, + author = {Allan Rechtschaffen and Anthony Kales}, + publisher = {Brain Information Service/Brain Research Institute, University of California}, + series = {Publication (National Institutes of Health (U.S.)) }, + title = {A Manual of Standardized Terminology, Techniques and Scoring System for Sleep Stages of Human Subjects}, + year = {1968} +} +@misc{AASM2007, + author = {Schulz, Hartmut}, + booktitle = {Journal of Clinical Sleep Medicine}, + file = {:Users/haodong/Documents/Papers/2007/Schulz - 2007 - The AASM manual for the scoring of sleep and associated events.pdf:pdf}, + isbn = {096572204X}, + issn = {15509389}, + mendeley-groups = {EEG/sleep classification}, + number = {7}, + pages = {752}, + pmid = {1559547}, + title = {{The AASM manual for the scoring of sleep and associated events}}, + volume = {3}, + year = {2007} +} +% Existing sleep stage classification method +@article{Orestis2015, + author = {Tsinalis, Orestis and Matthews, Paul M. and Guo, Yike}, + doi = {10.1007/s10439-015-1444-y}, + file = {:Users/haodong/Documents/Papers/2015/Tsinalis, Matthews, Guo - 2015 - Automatic Sleep Stage Scoring Using Time-Frequency Analysis and Stacked Sparse Autoencoders.pdf:pdf}, + isbn = {10.1007/s10439-015-1444-y}, + issn = {15739686}, + journal = {Annals of Biomedical Engineering}, + keywords = {Deep learning,EEG,Electroencephalography,Ensemble learning}, + mendeley-groups = {EEG/sleep classification}, + pmid = {26464268}, + title = {{Automatic Sleep Stage Scoring Using Time-Frequency Analysis and Stacked Sparse Autoencoders}}, + year = {2015} +} +@article{Fraiwan2012, + author = {Fraiwan, Luay and Lweesy, Khaldon and Khasawneh, Natheer and Wenz, Heinrich and Dickhaus, Hartmut}, + doi = {10.1016/j.cmpb.2011.11.005}, + file = {:Users/haodong/Documents/Papers/2012/Fraiwan et al. - 2012 - Automated sleep stage identification system based on time-frequency analysis of a single EEG channel and random.pdf:pdf}, + isbn = {1872-7565 (Electronic)$\backslash$r0169-2607 (Linking)}, + issn = {01692607}, + journal = {Computer Methods and Programs in Biomedicine}, + keywords = {Automated sleep stage identification,Choi-Williams distribution (CWD),Continuous wavelet transform (CWT),Hilbert-Huang Transform (HHT),Random forest classifier,Time-frequency analysis}, + number = {1}, + pages = {10--19}, + pmid = {22178068}, + publisher = {Elsevier Ireland Ltd}, + title = {{Automated sleep stage identification system based on time-frequency analysis of a single EEG channel and random forest classifier}}, + volume = {108}, + year = {2012} +} +@article{Liang2012, + author = {Liang, Sheng Fu and Kuo, Chin En and Hu, Yu Han and Pan, Yu Hsiang and Wang, Yung Hung}, + doi = {10.1109/TIM.2012.2187242}, + file = {:Users/haodong/Documents/Papers/2012/Liang et al. - 2012 - Automatic stage scoring of single-channel sleep EEG by using multiscale entropy and autoregressive models.pdf:pdf}, + issn = {00189456}, + journal = {IEEE Transactions on Instrumentation and Measurement}, + keywords = {Automatic sleep scoring,autoregressive (AR) model,linear discriminant analysis (LDA),multiscale entropy (MSE),single-channel electroencephalogram (EEG)}, + mendeley-groups = {EEG/sleep classification}, + number = {6}, + pages = {1649--1657}, + title = {{Automatic stage scoring of single-channel sleep EEG by using multiscale entropy and autoregressive models}}, + volume = {61}, + year = {2012} +} +@article{Berthomier2007, + author = {Berthomier, Christian and Drouot, Xavier and Herman-Sto{\"{i}}ca, Maria and Berthomier, Pierre and Prado, Jacques and Bokar-Thire, Djibril and Benoit, Odile and Mattout, J{\'{e}}r{\'{e}}mie and D'Ortho, Marie-Pia}, + file = {:Users/haodong/Documents/Papers/2007/Berthomier et al. - 2007 - Automatic analysis of single-channel sleep EEG validation in healthy individuals.pdf:pdf}, + isbn = {0161-8105 (Print)$\backslash$n0161-8105 (Linking)}, + issn = {0161-8105}, + journal = {Sleep}, + keywords = {automatic analysis,automatic sleep scoring,benoit o,berthomier c,berthomier p,bokar-thire d,citation,clinical valida-,d,do j,drouot x,eeg,healthy subjects,herman-sto{\"{i}}ca m,mattout j,ortho mp,pra-,single channel,tion}, + mendeley-groups = {EEG/sleep classification}, + number = {11}, + pages = {1587--1595}, + pmid = {18041491}, + title = {{Automatic analysis of single-channel sleep EEG: validation in healthy individuals.}}, + volume = {30}, + year = {2007} +} +% EEG electrode +@article{ChiDryElectrodeReview2010, + author = {Chi, Yu Mike and Jung, Tzyy-ping and Cauwenberghs, Gert}, + doi = {10.1109/RBME.2010.2084078}, + isbn = {1937-3333}, + issn = {1941-1189}, + journal = {IEEE reviews in biomedical engineering}, + mendeley-groups = {EEG/dry electrode}, + pages = {106--119}, + pmid = {22275204}, + title = {{Dry-contact and Non-contact Biopotential}}, + volume = {3}, + year = {2010} +} +% Open access data +@misc{SleepEDFX-OrestisUsed, + author = {PhysioNet}, + title = {{The Sleep-EDF database [Expanded]}}, + howpublished = "\url{http:// www.physionet.org/physiobank/database/sleep-edfx/}", + year = {2015}, + % note = "[Online; accessed 19-July-2008]" +} +@article{MASSData2014, + author = {O'Reilly, Christian and Gosselin, Nadia and Carrier, Julie and Nielsen, Tore}, + doi = {10.1111/jsr.12169}, + isbn = {1365-2869 (Electronic)$\backslash$r0962-1105 (Linking)}, + issn = {13652869}, + journal = {Journal of Sleep Research}, + keywords = {Collaboration tool,Methodology,Open-data,Reproducibility}, + mendeley-groups = {EEG/sleep classification}, + number = {6}, + pages = {628--635}, + pmid = {24909981}, + title = {{Montreal archive of sleep studies: An open-access resource for instrument benchmarking and exploratory research}}, + volume = {23}, + year = {2014} +} + +%% Sleep scoring using Forehead +@article{levendowski2012retrospective, + title={Retrospective cross-validation of automated sleep staging using electroocular recording in patients with and without sleep disordered breathing}, + author={Levendowski, Daniel J and Popovic, Djordje and Berka, Chris and Westbrook, Philip R}, + journal={International archives of medicine}, + volume={5}, + number={1}, + pages={21}, + year={2012}, + publisher={BioMed Central} +} +@book{popovic2011dissertation, + title={Automatic staging of sleep using only two electrodes on the forehead}, + author={Popovic, Djordje}, + year={2011}, + publisher={University of Southern California} +} +@article{popovic2014automatic, + title={Automatic scoring of sleep stages and cortical arousals using two electrodes on the forehead: validation in healthy adults}, + author={Popovic, Djordje and Khoo, Michael and Westbrook, Philip}, + journal={Journal of sleep research}, + volume={23}, + number={2}, + pages={211--221}, + year={2014}, + publisher={Wiley Online Library} +} +@article{virkkala2007automatic, + title={Automatic sleep stage classification using two-channel electro-oculography}, + author={Virkkala, Jussi and Hasan, Joel and V{\"a}rri, Alpo and Himanen, Sari-Leena and M{\"u}ller, Kiti}, + journal={Journal of neuroscience methods}, + volume={166}, + number={1}, + pages={109--115}, + year={2007}, + publisher={Elsevier} +} +@article{poree2006, + isbn = {1089-7771 (Print)$\backslash$n1089-7771 (Linking)}, + issn = {10897771}, + journal = {IEEE Transactions on Information Technology in Biomedicine}, + keywords = {Ambulatory recording,Blind source separation,Electrocardiogram (ECG),Electroencephalogram (EEG),Electromyogram (EMG),Electrooculogram (EOG),Home recording,Independent component analysis (ICA),Polysomnography,Sleep analysis,Sleep apnea syndrome (SAS),Sleep disorders,Sleep staging}, + number = {2}, + pages = {293--301}, + pmid = {16617618}, + title = {{Blind source separation for ambulatory sleep recording}}, + volume = {10}, + year = {2006} +} +@inproceedings{huang2013hierarchical, + title={A hierarchical classification system for sleep stage scoring via forehead EEG signals}, + author={Huang, Chihsheng and Lin, Chunling and Ko, Liwei and Liu, Shengyi and Sua, Tungping and Lin, Chinteng}, + booktitle={Computational Intelligence, Cognitive Algorithms, Mind, and Brain (CCMB), 2013 IEEE Symposium on}, + pages={1--5}, + year={2013}, + organization={IEEE} +} +@article{stepnowsky2013scoring, + title={Scoring accuracy of automated sleep staging from a bipolar electroocular recording compared to manual scoring by multiple raters}, + author={Stepnowsky, Carl and Levendowski, Daniel and Popovic, Djordje and Ayappa, Indu and Rapoport, David M}, + journal={Sleep medicine}, + volume={14}, + number={11}, + pages={1199--1207}, + year={2013}, + publisher={Elsevier} +} + % SGD +@inproceedings{bottouSGD2007, + author = {Bottou, L and Bousquet, Olivier}, + isbn = {160560352X}, + issn = {{\textless}null{\textgreater}}, + booktitle = {Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + mendeley-groups = {Machine Learning/training methods}, + pages = {161--168}, + title = {{The Tradeoffs of Large Scale Learning.}}, + volume = {20}, + year = {2007} +} +@inproceedings{le2011optimization, + title={On optimization methods for deep learning}, + author={Le, Quoc V and Ngiam, Jiquan and Coates, Adam and Lahiri, Abhik and Prochnow, Bobby and Ng, Andrew Y}, + booktitle={Proceedings of the 28th International Conference on International Conference on Machine Learning}, + pages={265--272}, + year={2011}, + organization={Omnipress} +} +%% SVM, MLP, KNN comparsion +@article{wu2004svm, + title={Probability estimates for multi-class classification by pairwise coupling}, + author={Wu, Ting-Fan and Lin, Chih-Jen and Weng, Ruby C}, + journal={{Journal of Machine Learning Research (JMLR)}}, + volume={5}, + number={Aug}, + pages={975--1005}, + year={2004} +} % MLP +@article{ruck1990multilayer, + title={The multilayer perceptron as an approximation to a Bayes optimal discriminant function}, + author={Ruck, Dennis W and Rogers, Steven K and Kabrisky, Matthew and Oxley, Mark E and Suter, Bruce W}, + journal={IEEE Transactions on Neural Networks}, + volume={1}, + number={4}, + pages={296--298}, + year={1990}, + publisher={IEEE} +} +@article{rosenblatt1958perceptron, + title={The perceptron: a probabilistic model for information storage and organization in the brain.}, + author={Rosenblatt, Frank}, + journal={Psychological Review}, + volume={65}, + number={6}, + pages={386}, + year={1958}, + publisher={American Psychological Association} +} +% K-means +@article{hartigan1975clustering, + title={Clustering algorithms}, + author={Hartigan, John A}, + year={1975}, + journal={Mathematical Physics and Mathematics}, + publisher={Wiley} +} +@article{cover1967nearest, + title={Nearest neighbor pattern classification}, + author={Cover, Thomas and Hart, Peter}, + journal={IEEE Transactions on Information Theory}, + volume={13}, + number={1}, + pages={21--27}, + year={1967}, + publisher={IEEE} +} +@article{breiman2001random, + title={Random forests}, + author={Breiman, Leo}, + journal={Machine Learning}, + volume={45}, + number={1}, + pages={5--32}, + year={2001}, + publisher={Springer} +} +@inproceedings{caruana2006empirical, + title={An empirical comparison of supervised learning algorithms}, + author={Caruana, Rich and Niculescu-Mizil, Alexandru}, + booktitle={Proceedings of the 23rd international conference on Machine learning}, + pages={161--168}, + year={2006}, + organization={ACM} +} +@article{SabetiEntropy2009, + author = {Sabeti, Malihe and Katebi, Serajeddin and Boostani, Reza}, + doi = {10.1016/j.artmed.2009.03.003}, + file = {:Users/haodong/Documents/Papers/2009/Sabeti, Katebi, Boostani - 2009 - Entropy and complexity measures for EEG signal classification of schizophrenic and control participant.pdf:pdf}, + isbn = {1873-2860 (Electronic) 1873-2860 (Linking)}, + issn = {09333657}, + journal = {Artificial Intelligence in Medicine}, + keywords = {Complexity,EEG classification,Entropy,Features selection,Schizophrenic}, + mendeley-groups = {Temp,EEG/REM SEM K-comp}, + number = {3}, + pages = {263--274}, + pmid = {19403281}, + title = {{Entropy and complexity measures for EEG signal classification of schizophrenic and control participants}}, + volume = {47}, + year = {2009} +} +@incollection{jolliffe2011pca, + title={Principal component analysis}, + author={Jolliffe, Ian}, + booktitle={International Encyclopedia of Statistical Science}, + pages={1094--1096}, + year={2011}, + publisher={Springer} +} +% EEG Feature detection +@article{RMEDetection, + author = {Yetton, Benjamin D and Niknazar, Mohammad and Duggan, Katherine A and McDevitt, Elizabeth A and Whitehurst, Lauren N and Sattari, Negin and Mednick, Sara C}, + doi = {10.1016/j.jneumeth.2015.11.015}, + file = {:Users/haodong/Documents/Papers/2015/Yetton et al. - 2015 - Automatic Detection of Rapid Eye Movements (REMs) A machine learning approach.pdf:pdf}, + issn = {1872-678X}, + journal = {Journal of neuroscience methods}, + mendeley-groups = {EEG/REM SEM K-comp}, + pages = {72--82}, + pmid = {26642967}, + publisher = {Elsevier B.V.}, + title = {{Automatic Detection of Rapid Eye Movements (REMs): A machine learning approach.}}, + volume = {259}, + year = {2015} +} +@article{SEMDetection, + author = {Cona, Filippo and Pizza, Fabio and Provini, Federica and Magosso, Elisa}, + doi = {10.1016/j.medengphy.2014.03.019}, + file = {:Users/haodong/Documents/Papers/2014/Cona et al. - 2014 - An improved algorithm for the automatic detection and characterization of slow eye movements.pdf:pdf}, + issn = {1873-4030}, + journal = {Medical engineering {\&} physics}, + keywords = {biomedical signal processing,sems,slow eye movements}, + mendeley-groups = {EEG/REM SEM K-comp}, + number = {7}, + pages = {954--61}, + pmid = {24768562}, + publisher = {Institute of Physics and Engineering in Medicine}, + title = {{An improved algorithm for the automatic detection and characterization of slow eye movements.}}, + volume = {36}, + year = {2014} +} +@book{PSGforSleepBook, + author = {Bonnie Robertson and Buddy Marshall and Margaret-Ann Carno}, + mendeley-groups = {EEG/sleep classification}, + publisher = {Mosby}, + series = {1 Edition}, + title = {{Polysomnography for the Sleep Technologist: Instrumentation, Monitoring, and Related Procedures}}, + page = {134-135}, + year = {2013} +} +@article{BandarabadiEpilepsy2014, + author = {Bandarabadi, Mojtaba and Teixeira, Cesar A. and Rasekhi, Jalil and Dourado, Antonio}, + doi = {10.1016/j.clinph.2014.05.022}, + isbn = {1872-8952 (Electronic)$\backslash$r1388-2457 (Linking)}, + issn = {18728952}, + journal = {Clinical Neurophysiology}, + keywords = {Classification,Epileptic seizure prediction,Feature reduction,Relative spectral power}, + mendeley-groups = {EEG/epilepsy}, + number = {2}, + pages = {237--248}, + pmid = {24969376}, + publisher = {International Federation of Clinical Neurophysiology}, + title = {{Epileptic seizure prediction using relative spectral power features}}, + volume = {126}, + year = {2015} +} + +%% EEGBook +@article{CohenBook2014, + author = {Mike X Cohen}, + title = {{Analyzing Neural Time Series Data: Theory and Practice}}, + publisher = {The MIT Press}, + year = {2014} +} +@article{bailey1994fast, + title={A fast method for the numerical evaluation of continuous Fourier and Laplace transforms}, + author={Bailey, David H and Swarztrauber, Paul N}, + journal={SIAM Journal on Scientific Computing}, + volume={15}, + number={5}, + pages={1105--1110}, + year={1994}, + publisher={SIAM} +} + +@article{harris1978use, + title={On the use of windows for harmonic analysis with the discrete Fourier transform}, + author={Harris, Fredric J}, + journal={Proceedings of the IEEE}, + volume={66}, + number={1}, + pages={51--83}, + year={1978}, + publisher={IEEE} +} + + +%% hao dong paper my paper +@inproceedings{dong2017automatic, + title={Automatic brain tumor detection and segmentation using u-net based fully convolutional networks}, + author={Dong, Hao and Yang, Guang and Liu, Fangde and Mo, Yuanhan and Guo, Yike}, + booktitle={Proceedings of the Annual Conference on Medical Image Understanding and Analysis (MIUA)}, + pages={506--517}, + year={2017}, + organization={Springer} +} +@inproceedings{tensorlayer2017, + author = {Dong, Hao and Supratak, Akara and Mai, Luo and Liu, Fangde and Oehmichen, Axel and Yu, Simiao and Guo, Yike}, + booktitle = {Proceedings of the ACM Multimedia (MM)}, + title = {{TensorLayer: a versatile library for efficient deep learning development}}, + url = {http://tensorlayer.org}, + year = {2017} +} +@book{tensorlayerbook2018, + author = {Dong, Hao and Yike, Guo and Guang, Yang}, + booktitle = {Publishing House of Electronics Industry}, + title = {Deep Learning using TensorLayer}, + year = {2018} +} +@inproceedings{liutensordb, + title={TensorDB: Database Infrastructure for Continuous Machine Learning}, + booktitle = {Proceedings of the International Conference on Artificial Intelligence (ICAI)}, + author={Liu, F and Oehmichen, A and Zhang, J and Sun, K and Dong, H and Mo, Y and Guo, Y}, + year = {2017} +} +@article{dong2017unsupervised, + title={Unsupervised image-to-image translation with generative adversarial networks}, + author={Dong, Hao and Neekhara, Paarth and Wu, Chao and Guo, Yike}, + journal={arXiv preprint arXiv:1701.02676}, + year={2017} +} +@article{dong2018mixed, + title={Mixed neural network approach for temporal sleep stage classification}, + author={Dong, Hao and Supratak, Akara and Pan, Wei and Wu, Chao and Matthews, Paul M and Guo, Yike}, + journal={IEEE Transactions on Neural Systems and Rehabilitation Engineering (TNSRE)}, + volume={26}, + number={2}, + pages={324--333}, + year={2018}, + publisher={IEEE} +} +@article{supratak2017deepsleepnet, + title={DeepSleepNet: A model for automatic sleep stage scoring based on raw single-channel EEG}, + author={Supratak, Akara and Dong, Hao and Wu, Chao and Guo, Yike}, + journal={IEEE Transactions on Neural Systems and Rehabilitation Engineering (TNSRE)}, + volume={25}, + number={11}, + pages={1998--2008}, + year={2017}, + publisher={IEEE} +} +@incollection{supratak2016survey, + title={Survey on feature extraction and applications of biosignals}, + author={Supratak, Akara and Wu, Chao and Dong, Hao and Sun, Kai and Guo, Yike}, + booktitle={Machine Learning for Health Informatics}, + pages={161--182}, + year={2016}, + publisher={Springer} +} +@inproceedings{suprataktowards, + title={Towards Desynchronization Detection in Biosignals}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop}, + author={Supratak, Akara and Schneider, Steffen and Dong, Hao and Li, Ling and Guo, Yike}, + year={2017}, +} +@inproceedings{dong2016new, + title={A new soft material based in-the-ear EEG recording technique}, + author={Dong, Hao and Matthews, Paul M and Guo, Yike}, + booktitle={Proceedings of the International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)}, + pages={5709--5712}, + year={2016}, + organization={IEEE} +} +@article{pan2016dropneuron, + title={DropNeuron: Simplifying the Structure of Deep Neural Networks}, + author={Pan, Wei and Dong, Hao and Guo, Yike}, + journal={arXiv preprint arXiv:1606.07326}, + year={2016} +} +@inproceedings{dong2017semantic, + title={Semantic image synthesis via adversarial learning}, + author={Dong, Hao and Yu, Simiao and Wu, Chao and Guo, Yike}, + booktitle={Proceedings of International Conference on Computer Vision (ICCV)}, + year={2017} +} +@article{yu2017deep, + title={{Deep de-aliasing for fast compressive sensing MRI}}, + author={Yu, Simiao and Dong, Hao and Yang, Guang and Slabaugh, Greg and Dragotti, Pier Luigi and Ye, Xujiong and Liu, Fangde and Arridge, Simon and Keegan, Jennifer and Firmin, David and others}, + journal={arXiv preprint arXiv:1705.07137}, + year={2017} +} +@article{yang2017dagan, + title={{DAGAN: Deep de-aliasing generative adversarial networks for fast compressed sensing MRI reconstruction}}, + author={Yang, Guang and Yu, Simiao and Dong, Hao and Slabaugh, Greg and Dragotti, Pier Luigi and Ye, Xujiong and Liu, Fangde and Arridge, Simon and Keegan, Jennifer and Guo, Yike and David Firmin}, + journal={IEEE Transactions on Medical Imaging (TMI)}, + year={2017}, + publisher={IEEE} +} +@inproceedings{dong2017i2t2i, + title={I2T2I: Learning text to image synthesis with textual data augmentation}, + author={Dong, Hao and Zhang, Jingqing and McIlwraith, Douglas and Guo, Yike}, + booktitle={Proceedings of the IEEE International Conference on Image Processing (ICIP)}, + year={2017} +} +@article{yu2018generative, + title={Generative Creativity: Adversarial Learning for Bionic Design}, + author={Yu, Simiao and Dong, Hao and Wang, Pan and Wu, Chao and Guo, Yike}, + journal={arXiv preprint arXiv:1805.07615}, + year={2018} +} +@article{dong2018dropping, + title={Dropping Activation Outputs with Localized First-layer Deep Network for Enhancing User Privacy and Data Security}, + author={Dong, Hao and Wu, Chao and Wei, Zhen and Guo, Yike}, + journal={IEEE Transactions on Information Forensics and Security (TIFS)}, + volume={13}, + number={3}, + pages={662--670}, + year={2018}, + publisher={IEEE} +} +@inproceedings{supratak2017towards, + title={Towards desynchronization detection in biosignals}, + author={Supratak, Akara and Schneider, Steffen and Dong, Hao and Li, Ling and Guo, Yike}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workspace}, + year={2017} +} + + +%% text/sentence data augmentation +@inproceedings{janiszek2001data, + title={Data augmentation and language model adaptation}, + author={Janiszek, David and De Mori, Renato and Bechet, E}, + booktitle={International Conference on Acoustics, Speech, and Signal Processing. Proceedings (ICASSP)}, + volume={1}, + pages={549--552}, + year={2001}, + organization={IEEE} +} + +@inproceedings{ko2015audio, + title={Audio augmentation for speech recognition}, + author={Ko, Tom and Peddinti, Vijayaditya and Povey, Daniel and Khudanpur, Sanjeev}, + booktitle={Annual Conference of the International Speech Communication Association}, + year={2015} +} + +@article{zhang2015text, + title={Text understanding from scratch}, + author={Zhang, Xiang and LeCun, Yann}, + journal={arXiv preprint arXiv:1502.01710}, + year={2015} +} +@inproceedings{zhang2015textAdvances in Neural Information Processing Systems, + title={Character-level convolutional networks for text classification}, + author={Zhang, Xiang and Zhao, Junbo and LeCun, Yann}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Conference}, + pages={649--657}, + year={2015} +} +@inproceedings{saito2017improving, + title={Improving Neural Text Normalization with Data Augmentation at Character-and Morphological Levels}, + author={Saito, Itsumi and Suzuki, Jun and Nishida, Kyosuke and Sadamitsu, Kugatsu and Kobashikawa, Satoshi and Masumura, Ryo and Matsumoto, Yuji and Tomita, Junji}, + booktitle={Proceedings of the International Joint Conference on Natural Language Processing}, + volume={2}, + pages={257--262}, + year={2017} +} + +%% NLP +@inproceedings{bahdanau2014neural, + title={Neural machine translation by jointly learning to align and translate}, + author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua}, + booktitle={Proceedings of the International Conference on Learning Representations (ICLR)}, + year={2015} +} + +% TensorLayer citation +@inproceedings{ju2019single, + title={Single-image super-resolution reconstruction via generative adversarial network}, + author={Ju, Chunwu and Su, Xiuqin and Yang, Haoyuan and Ning, Hailong}, + booktitle={9th International Symposium on Advanced Optical Manufacturing and Testing Technologies: Optoelectronic Materials and Devices for Sensing and Imaging}, + volume={10843}, + pages={108430J}, + year={2019}, + organization={International Society for Optics and Photonics} +} +@article{mukherjee2018convolutional, + title={Convolutional neural networks for whole slide image superresolution}, + author={Mukherjee, Lopamudra and Keikhosravi, Adib and Bui, Dat and Eliceiri, Kevin W}, + journal={Biomedical Optics Express}, + volume={9}, + number={11}, + pages={5368--5386}, + year={2018}, + publisher={Optical Society of America} +} +@article{wu2019exploratory, + title={Exploratory Analysis for Big Social Data Using Deep Network}, + author={Wu, Chao and Wang, Guolong and Zhu, Jiangcheng and Lertvittayakumjorn, Piyawat and Hu, Simon and Tan, Chilie and Mi, Hong and Xu, Yadan and Xiao, Jun}, + journal={IEEE Access}, + year={2019}, + publisher={IEEE} +} +@article{fan2019two, + title={Two-stream siamese network with contrastive-center losses for RGB-D action recognition}, + author={Fan, Chunxiao and Zhai, Zhengyuan and Ming, Yue and Tian, Lei}, + journal={Journal of Electronic Imaging}, + volume={28}, + number={2}, + pages={023004}, + year={2019}, + publisher={International Society for Optics and Photonics} +} + +%% shuffle +@book{fisher1943statistical, + title={Statistical tables for biological, agricultural and medical research}, + author={Fisher, Ronald A and Yates, Frank}, + year={1943}, + publisher={Oliver and Boyd Ltd, London} +} + +%% used in Policy Iteration +@techreport{williams1993analysis, + title={Analysis of some incremental variants of policy iteration: First steps toward understanding actor-critic learning systems}, + author={Williams, Ronald J and Baird III, Leemon C}, + year={1993}, + institution={Tech. rep. NU-CCS-93-11, Northeastern University, College of Computer Science} +} + +@inproceedings{tsitsiklis1997analysis, + title={Analysis of temporal-diffference learning with function approximation}, + author={Tsitsiklis, John N and Van Roy, Benjamin}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1075--1081}, + year={1997} +} + +@article{mcclelland1995there, + title={Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.}, + author={McClelland, James L and McNaughton, Bruce L and O'Reilly, Randall C}, + journal={Psychological review}, + volume={102}, + number={3}, + pages={419}, + year={1995}, + publisher={American Psychological Association} +} + +@article{o2010play, + title={Play it again: reactivation of waking experience and memory}, + author={O'Neill, Joseph and Pleydell-Bouverie, Barty and Dupret, David and Csicsvari, Jozsef}, + journal={Trends in neurosciences}, + volume={33}, + number={5}, + pages={220--229}, + year={2010}, + publisher={Elsevier} +} + +@techreport{lin1993reinforcement, + title={Reinforcement learning for robots using neural networks}, + author={Lin, Long-Ji}, + year={1993}, + institution={Carnegie-Mellon Univ Pittsburgh PA School of Computer Science} +} + +@inproceedings{riedmiller2005neural, + title={Neural fitted {Q} iteration--first experiences with a data efficient neural reinforcement learning method}, + author={Riedmiller, Martin}, + booktitle={European Conference on Machine Learning}, + pages={317--328}, + year={2005}, + organization={Springer} +} + +@inproceedings{van2016deep, + title={Deep reinforcement learning with double {Q}-learning}, + author={Van Hasselt, Hado and Guez, Arthur and Silver, David}, + booktitle={Thirtieth AAAI conference on artificial intelligence}, + year={2016} +} + +@inproceedings{wang2016dueling, + title={Dueling Network Architectures for Deep Reinforcement Learning}, + author={Wang, Ziyu and Schaul, Tom and Hessel, Matteo and Hasselt, Hado and Lanctot, Marc and Freitas, Nando}, + booktitle={International Conference on Machine Learning}, + pages={1995--2003}, + year={2016} +} + +@inproceedings{schaul2015prioritized, + title={Prioritized experience replay}, + author={Schaul, Tom and Quan, John and Antonoglou, Ioannis and Silver, David}, + booktitle={arXiv preprint arXiv:1511.05952}, + year={2015} +} + +% @article{schaul2015prioritized, +% title={Prioritized experience replay}, +% author={Schaul, Tom and Quan, John and Antonoglou, Ioannis and Silver, David}, +% journal={arXiv preprint arXiv:1511.05952}, +% year={2015} +% } + +% @ARTICLE{schaul2015prioritized, +% author = {{Schaul}, Tom and {Quan}, John and {Antonoglou}, Ioannis and +% {Silver}, David}, +% title = "{Prioritized Experience Replay}", +% journal = {arXiv e-prints}, +% keywords = {Computer Science - Machine Learning}, +% year = 2015, +% month = nov, +% eid = {arXiv:1511.05952}, +% pages = {arXiv:1511.05952}, +% archivePrefix = {arXiv}, +% eprint = {1511.05952}, +% primaryClass = {cs.LG}, +% adsurl = {https://ui.adsabs.harvard.edu/abs/2015arXiv151105952S}, +% adsnote = {Provided by the SAO/NASA Astrophysics Data System} +% } + +@inproceedings{hessel2018rainbow, + title={Rainbow: Combining improvements in deep reinforcement learning}, + author={Hessel, Matteo and Modayil, Joseph and Van Hasselt, Hado and Schaul, Tom and Ostrovski, Georg and Dabney, Will and Horgan, Dan and Piot, Bilal and Azar, Mohammad and Silver, David}, + booktitle={Thirty-Second AAAI Conference on Artificial Intelligence}, + year={2018} +} + +@article{castro18dopamine, + author={Pablo Samuel Castro and Subhodeep Moitra and Carles Gelada and Saurabh Kumar and Marc G. Bellemare}, + title={Dopamine: A Research Framework for Deep Reinforcement Learning}, + year={2018}, + url={http://arxiv.org/abs/1812.06110}, + archivePrefix = {arXiv} +} + +@inproceedings{hernandez2019understanding, + title={Understanding Multi-Step Deep Reinforcement Learning: A Systematic Study of the {DQN} Target}, + author={Hernandez-Garcia, J Fernando and Sutton, Richard S}, + booktitle={Proceedings of the Neural Information Processing Systems (Advances in Neural Information Processing Systems) Workshop}, + year={2019} +} + +@inproceedings{thrun1993issues, + title={Issues in using function approximation for reinforcement learning}, + author={Thrun, Sebastian and Schwartz, Anton}, + booktitle={Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum}, + year={1993} +} + +@misc{deepmind2015dqn, + author = {DeepMind}, + title={Lua/{Torch} implementation of {DQN}}, + year={2015}, + publisher = {GitHub}, + journal = {GitHub repository}, + howpublished = {\url{https://github.com/deepmind/dqn}}, +} + +@article{tsitsiklis1996analysis, + title={An analysis of temporal-difference learning with function approximationTechnical}, + author={Tsitsiklis, JN and Van Roy, B}, + journal={Report LIDS-P-2322). Laboratory for Information and Decision Systems, Massachusetts Institute of Technology, Tech. Rep.}, + year={1996} +} + +@incollection{huber1992robust, + title={Robust estimation of a location parameter}, + author={Huber, Peter J}, + booktitle={Breakthroughs in statistics}, + pages={492--518}, + year={1992}, + publisher={Springer} +} + +@inproceedings{szepesvari1998asymptotic, + title={The asymptotic convergence-rate of Q-learning}, + author={Szepesv{\'a}ri, Csaba}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1064--1070}, + year={1998} +} + + +@article{singh2000convergence, + title={Convergence results for single-step on-policy reinforcement-learning algorithms}, + author={Singh, Satinder and Jaakkola, Tommi and Littman, Michael L and Szepesv{\'a}ri, Csaba}, + journal={Machine learning}, + volume={38}, + number={3}, + pages={287--308}, + year={2000}, + publisher={Springer} +} + + +@book{sutton2018reinforcement, + title={Reinforcement learning: An introduction}, + author={Sutton, Richard S and Barto, Andrew G}, + year={2018}, + publisher={MIT press} +} + +@article{fortunato2017noisy, + title={Noisy networks for exploration}, + author={Fortunato, Meire and Azar, Mohammad Gheshlaghi and Piot, Bilal and Menick, Jacob and Osband, Ian and Graves, Alex and Mnih, Vlad and Munos, Remi and Hassabis, Demis and Pietquin, Olivier and others}, + journal={arXiv preprint arXiv:1706.10295}, + year={2017} +} + +@article{roderick2017implementing, + title={Implementing the deep {Q}-network}, + author={Roderick, Melrose and MacGlashan, James and Tellex, Stefanie}, + journal={arXiv preprint arXiv:1711.07478}, + year={2017} +} + +@inproceedings{bellemare2017distributional, + title={A distributional perspective on reinforcement learning}, + author={Bellemare, Marc G and Dabney, Will and Munos, R{\'e}mi}, + booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70}, + pages={449--458}, + year={2017}, + organization={JMLR. org} +} + +@inproceedings{yang2019fully, + title={Fully Parameterized Quantile Function for Distributional Reinforcement Learning}, + author={Yang, Derek and Zhao, Li and Lin, Zichuan and Qin, Tao and Bian, Jiang and Liu, Tie-Yan}, + booktitle={Advances in Neural Information Processing Systems}, + pages={6190--6199}, + year={2019} +} + +@inproceedings{mavrin2019distributional, + title={Distributional Reinforcement Learning for Efficient Exploration}, + author={Mavrin, Borislav and Yao, Hengshuai and Kong, Linglong and Wu, Kaiwen and Yu, Yaoliang}, + booktitle={International Conference on Machine Learning}, + pages={4424--4434}, + year={2019} +} + +@inproceedings{dabney2018distributional, + title={Distributional reinforcement learning with quantile regression}, + author={Dabney, Will and Rowland, Mark and Bellemare, Marc G and Munos, R{\'e}mi}, + booktitle={Thirty-Second AAAI Conference on Artificial Intelligence}, + year={2018} +} + +@inproceedings{dabney2018implicit, + title={Implicit Quantile Networks for Distributional Reinforcement Learning}, + author={Dabney, Will and Ostrovski, Georg and Silver, David and Munos, Remi}, + booktitle={International Conference on Machine Learning}, + pages={1104--1113}, + year={2018} +} + +@inproceedings{bychkovsky2011learning, + title={Learning photographic global tonal adjustment with a database of input/output image pairs}, + author={Bychkovsky, Vladimir and Paris, Sylvain and Chan, Eric and Durand, Fr{\'e}do}, + booktitle={CVPR 2011}, + pages={97--104}, + year={2011}, + organization={IEEE} +} + +@inproceedings{wang2019underexposed, + title={Underexposed Photo Enhancement Using Deep Illumination Estimation}, + author={Wang, Ruixing and Zhang, Qing and Fu, Chi-Wing and Shen, Xiaoyong and Zheng, Wei-Shi and Jia, Jiaya}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={6849--6857}, + year={2019} +} + +@inproceedings{ulyanov2018deep, + title={Deep image prior}, + author={Ulyanov, Dmitry and Vedaldi, Andrea and Lempitsky, Victor}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={9446--9454}, + year={2018} +} + +@inproceedings{kupyn2018deblurgan, + title={{DeblurGAN}: Blind motion deblurring using conditional adversarial networks}, + author={Kupyn, Orest and Budzan, Volodymyr and Mykhailych, Mykola and Mishkin, Dmytro and Matas, Ji{\v{r}}{\'\i}}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={8183--8192}, + year={2018} +} + +@article{zhang2019gan, + title={{GAN} Based Image Deblurring Using Dark Channel Prior}, + author={Zhang, Shuang and Zhen, Ada and Stevenson, Robert L}, + journal={arXiv preprint arXiv:1903.00107}, + year={2019} +} + +@article{mao2019tunagan, + title={Tuna{GAN}: Interpretable {GAN} for Smart Editing}, + author={Mao, Weiquan and Lou, Beicheng and Yuan, Jiyao}, + journal={arXiv preprint arXiv:1908.06163}, + year={2019} +} + +@article{nataraj2019detecting, + title={Detecting {GAN} generated fake images using co-occurrence matrices}, + author={Nataraj, Lakshmanan and Mohammed, Tajuddin Manhar and Manjunath, BS and Chandrasekaran, Shivkumar and Flenner, Arjuna and Bappy, Jawadul H and Roy-Chowdhury, Amit K}, + journal={Journal of Electronic Imaging}, + year={2019}, + publisher={International Society for Optics and Photonics} +} + +@inproceedings{park2018distort, + title={Distort-and-recover: Color enhancement using deep reinforcement learning}, + author={Park, Jongchan and Lee, Joon-Young and Yoo, Donggeun and So Kweon, In}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={5928--5936}, + year={2018} +} + +@book{poynton2012digital, + title={Digital video and {HD}: Algorithms and Interfaces}, + author={Poynton, Charles}, + year={2012}, + publisher={Elsevier} +} + +@article{van2014scikit, + title={scikit-image: image processing in Python}, + author={Van der Walt, Stefan and Sch{\"o}nberger, Johannes L and Nunez-Iglesias, Juan and Boulogne, Fran{\c{c}}ois and Warner, Joshua D and Yager, Neil and Gouillart, Emmanuelle and Yu, Tony}, + journal={PeerJ}, + volume={2}, + pages={e453}, + year={2014}, + publisher={PeerJ Inc.} +} + +@misc{clark2015pillow, + title={Pillow (PIL Fork) Documentation}, + author={Clark, Alex}, + year={2015}, + howpublished={\url{https://github.com/python-pillow/Pillow}} +} + +@inproceedings{ren2015faster, + title={Faster {R-CNN}: Towards real-time object detection with region proposal networks}, + author={Ren, Shaoqing and He, Kaiming and Girshick, Ross and Sun, Jian}, + booktitle={Advances in Neural Information Processing Systems}, + pages={91--99}, + year={2015} +} + +@inproceedings{yu2018crafting, + title={Crafting a toolchain for image restoration by deep reinforcement learning}, + author={Yu, Ke and Dong, Chao and Lin, Liang and Change Loy, Chen}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={2443--2452}, + year={2018} +} + +@inproceedings{furuta2019fully, + title={Fully convolutional network with multi-step reinforcement learning for image processing}, + author={Furuta, Ryosuke and Inoue, Naoto and Yamasaki, Toshihiko}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + volume={33}, + pages={3598--3605}, + year={2019} +} + +@article{lee2005evaluation, + title={Evaluation of image similarity by histogram intersection}, + author={Lee, SM and Xin, JH and Westland, S}, + journal={Color Research \& Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Fran{\c{c}}ais de la Couleur}, + volume={30}, + number={4}, + pages={265--274}, + year={2005}, + publisher={Wiley Online Library} +} + +@article{ren2016object, + title={Object detection networks on convolutional feature maps}, + author={Ren, Shaoqing and He, Kaiming and Girshick, Ross and Zhang, Xiangyu and Sun, Jian}, + journal={IEEE transactions on pattern analysis and machine intelligence}, + volume={39}, + number={7}, + pages={1476--1481}, + year={2016}, + publisher={IEEE} +} + +@inproceedings{redmon2016you, + title={You only look once: Unified, real-time object detection}, + author={Redmon, Joseph and Divvala, Santosh and Girshick, Ross and Farhadi, Ali}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={779--788}, + year={2016} +} + +@InProceedings{Nah_2017_CVPR, + author = {Nah, Seungjun and Kim, Tae Hyun and Lee, Kyoung Mu}, + title = {Deep Multi-Scale Convolutional Neural Network for Dynamic Scene Deblurring}, + booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)}, + month = {July}, + year = {2017} +} + + +% chap 08 imitation learning +@article{sun2019provably, + title={Provably Efficient Imitation Learning from Observation Alone}, + author={Sun, Wen and Vemula, Anirudh and Boots, Byron and Bagnell, J Andrew}, + journal={arXiv preprint arXiv:1905.10948}, + year={2019} +} + +@article{jeong2019self, + title={Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation}, + author={Jeong, Rae and Aytar, Yusuf and Khosid, David and Zhou, Yuxiang and Kay, Jackie and Lampe, Thomas and Bousmalis, Konstantinos and Nori, Francesco}, + journal={arXiv preprint arXiv:1910.09470}, + year={2019} +} + +@article{torabi2019recent, + title={Recent Advances in Imitation Learning from Observation}, + author={Torabi, Faraz and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1905.13566}, + year={2019} +} + +@inproceedings{nair2018overcoming, + title={Overcoming exploration in reinforcement learning with demonstrations}, + author={Nair, Ashvin and McGrew, Bob and Andrychowicz, Marcin and Zaremba, Wojciech and Abbeel, Pieter}, + booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={6292--6299}, + year={2018}, + organization={IEEE} +} + +@inproceedings{ng1999policy, + title={Policy invariance under reward transformations: Theory and application to reward shaping}, + author={Ng, Andrew Y and Harada, Daishi and Russell, Stuart}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + volume={99}, + pages={278--287}, + year={1999} +} + +@article{finn2016connection, + title={A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models}, + author={Finn, Chelsea and Christiano, Paul and Abbeel, Pieter and Levine, Sergey}, + journal={arXiv preprint arXiv:1611.03852}, + year={2016} +} + +@inproceedings{henderson2018deep, + title={Deep reinforcement learning that matters}, + author={Henderson, Peter and Islam, Riashat and Bachman, Philip and Pineau, Joelle and Precup, Doina and Meger, David}, + booktitle={Thirty-Second AAAI Conference on Artificial Intelligence}, + year={2018} +} + +@article{jeong2019modelling, + title={Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer}, + author={Jeong, Rae and Kay, Jackie and Romano, Francesco and Lampe, Thomas and Rothorl, Tom and Abdolmaleki, Abbas and Erez, Tom and Tassa, Yuval and Nori, Francesco}, + journal={arXiv preprint arXiv:1910.09471}, + year={2019} +} + +@misc{houthooft2016vime, + title={VIME: Variational Information Maximizing Exploration}, + author={Rein Houthooft and Xi Chen and Yan Duan and John Schulman and Filip De Turck and Pieter Abbeel}, + year={2016}, + eprint={1605.09674}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@article{fu2017learning, + title={Learning robust rewards with adversarial inverse reinforcement learning}, + author={Fu, Justin and Luo, Katie and Levine, Sergey}, + journal={arXiv preprint arXiv:1710.11248}, + year={2017} +} + +@article{jaquier2019learning, + title={Learning from demonstration with model-based Gaussian process}, + author={Jaquier, No{\'e}mie and Ginsbourger, David and Calinon, Sylvain}, + journal={arXiv preprint arXiv:1910.05005}, + year={2019} +} + +@inproceedings{schneider2010robot, + title={Robot learning by demonstration with local gaussian process regression}, + author={Schneider, Markus and Ertel, Wolfgang}, + booktitle={2010 IEEE/RSJ International Conference on Intelligent Robots and Systems}, + pages={255--260}, + year={2010}, + organization={IEEE} +} + +@article{huang2019kernelized, + title={Kernelized movement primitives}, + author={Huang, Yanlong and Rozo, Leonel and Silv{\'e}rio, Jo{\~a}o and Caldwell, Darwin G}, + journal={The International Journal of Robotics Research}, + volume={38}, + number={7}, + pages={833--852}, + year={2019}, + publisher={SAGE Publications Sage UK: London, England} +} + +@inproceedings{paraschos2013probabilistic, + title={Probabilistic movement primitives}, + author={Paraschos, Alexandros and Daniel, Christian and Peters, Jan R and Neumann, Gerhard}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2616--2624}, + year={2013} +} + +@inproceedings{pastor2009learning, + title={Learning and generalization of motor skills by learning from demonstration}, + author={Pastor, Peter and Hoffmann, Heiko and Asfour, Tamim and Schaal, Stefan}, + booktitle={2009 IEEE International Conference on Robotics and Automation}, + pages={763--768}, + year={2009}, + organization={IEEE} +} + +@article{calinon2016tutorial, + title={A tutorial on task-parameterized movement learning and retrieval}, + author={Calinon, Sylvain}, + journal={Intelligent Service Robotics}, + volume={9}, + number={1}, + pages={1--29}, + year={2016}, + publisher={Springer} +} + +@article{sieb2019graph, + title={Graph-Structured Visual Imitation}, + author={Sieb, Maximilian and Xian, Zhou and Huang, Audrey and Kroemer, Oliver and Fragkiadaki, Katerina}, + journal={arXiv preprint arXiv:1907.05518}, + year={2019} +} + +@article{vevcerik2017leveraging, + title={Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards}, + author={Ve{\v{c}}er{\'\i}k, Matej and Hester, Todd and Scholz, Jonathan and Wang, Fumin and Pietquin, Olivier and Piot, Bilal and Heess, Nicolas and Roth{\"o}rl, Thomas and Lampe, Thomas and Riedmiller, Martin}, + journal={arXiv preprint arXiv:1707.08817}, + year={2017} +} + +@inproceedings{machado2015domain, + title={Domain-independent optimistic initialization for reinforcement learning}, + author={Machado, Marlos C and Srinivasan, Sriram and Bowling, Michael}, + booktitle={Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence}, + year={2015} +} +@article{sermanet2016unsupervised, + title={Unsupervised perceptual rewards for imitation learning}, + author={Sermanet, Pierre and Xu, Kelvin and Levine, Sergey}, + journal={arXiv preprint arXiv:1612.06699}, + year={2016} +} + +@inproceedings{brys2015reinforcement, + title={Reinforcement learning from demonstration through shaping}, + author={Brys, Tim and Harutyunyan, Anna and Suay, Halit Bener and Chernova, Sonia and Taylor, Matthew E and Now{\'e}, Ann}, + booktitle={Twenty-Fourth International Joint Conference on Artificial Intelligence}, + year={2015} +} + + +@article{gao2018reinforcement, + title={Reinforcement learning from imperfect demonstrations}, + author={Gao, Yang and Lin, Ji and Yu, Fisher and Levine, Sergey and Darrell, Trevor and others}, + journal={arXiv preprint arXiv:1802.05313}, + year={2018} +} + + +@inproceedings{hester2018deep, + title={Deep {Q}-learning from demonstrations}, + author={Hester, Todd and Vecerik, Matej and Pietquin, Olivier and Lanctot, Marc and Schaul, Tom and Piot, Bilal and Horgan, Dan and Quan, John and Sendonaris, Andrew and Osband, Ian and others}, + booktitle={Thirty-Second AAAI Conference on Artificial Intelligence}, + year={2018} +} + +@inproceedings{liu2018imitation, + title={Imitation from observation: Learning to imitate behaviors from raw video via context translation}, + author={Liu, YuXuan and Gupta, Abhishek and Abbeel, Pieter and Levine, Sergey}, + booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={1118--1125}, + year={2018}, + organization={IEEE} +} + +@article{sun2019adversarial, + title={Adversarial Imitation Learning from Incomplete Demonstrations}, + author={Sun, Mingfei and Ma, Xiaojuan}, + journal={arXiv preprint arXiv:1905.12310}, + year={2019} +} + +@article{sun2019provably, + title={Provably Efficient Imitation Learning from Observation Alone}, + author={Sun, Wen and Vemula, Anirudh and Boots, Byron and Bagnell, J Andrew}, + journal={arXiv preprint arXiv:1905.10948}, + year={2019} +} + +@article{stadie2017third, + title={Third-person imitation learning}, + author={Stadie, Bradly C and Abbeel, Pieter and Sutskever, Ilya}, + journal={arXiv preprint arXiv:1703.01703}, + year={2017} +} + +@article{zolna2018reinforced, + title={Reinforced Imitation Learning from Observations}, + author={Zo{\l}na, Konrad and Rostamzadeh, Negar and Bengio, Yoshua and Ahn, Sungjin and Pinheiro, Pedro O}, + year={2018} +} + +@inproceedings{tassa2012synthesis, + title={Synthesis and stabilization of complex behaviors through online trajectory optimization}, + author={Tassa, Yuval and Erez, Tom and Todorov, Emanuel}, + booktitle={2012 IEEE/RSJ International Conference on Intelligent Robots and Systems}, + pages={4906--4913}, + year={2012}, + organization={IEEE} +} + +@article{torabi2019sample, + title={Sample-efficient Adversarial Imitation Learning from Observation}, + author={Torabi, Faraz and Geiger, Sean and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1906.07374}, + year={2019} +} + +@article{torabi2019imitation, + title={Imitation Learning from Video by Leveraging Proprioception}, + author={Torabi, Faraz and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1905.09335}, + year={2019} +} + +@article{torabi2018generative, + title={Generative adversarial imitation from observation}, + author={Torabi, Faraz and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1807.06158}, + year={2018} +} + +@inproceedings{torabi2019adversarial, + title={Adversarial imitation learning from state-only demonstrations}, + author={Torabi, Faraz and Warnell, Garrett and Stone, Peter}, + booktitle={Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems}, + pages={2229--2231}, + year={2019}, + organization={International Foundation for Autonomous Agents and Multiagent Systems} +} + +@inproceedings{henderson2018optiongan, + title={Option{GAN}: Learning joint reward-policy options using generative adversarial inverse reinforcement learning}, + author={Henderson, Peter and Chang, Wei-Di and Bacon, Pierre-Luc and Meger, David and Pineau, Joelle and Precup, Doina}, + booktitle={Thirty-Second AAAI Conference on Artificial Intelligence}, + year={2018} +} + +@article{merel2017learning, + title={Learning human behaviors from motion capture by adversarial imitation}, + author={Merel, Josh and Tassa, Yuval and Srinivasan, Sriram and Lemmon, Jay and Wang, Ziyu and Wayne, Greg and Heess, Nicolas}, + journal={arXiv preprint arXiv:1707.02201}, + year={2017} +} + +@article{gupta2017learning, + title={Learning invariant feature spaces to transfer skills with reinforcement learning}, + author={Gupta, Abhishek and Devin, Coline and Liu, YuXuan and Abbeel, Pieter and Levine, Sergey}, + journal={arXiv preprint arXiv:1703.02949}, + year={2017} +} + +@inproceedings{aytar2018playing, + title={Playing hard exploration games by watching youtube}, + author={Aytar, Yusuf and Pfaff, Tobias and Budden, David and Paine, Thomas and Wang, Ziyu and de Freitas, Nando}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2930--2941}, + year={2018} +} + +@inproceedings{misra2016shuffle, + title={Shuffle and learn: unsupervised learning using temporal order verification}, + author={Misra, Ishan and Zitnick, C Lawrence and Hebert, Martial}, + booktitle={European Conference on Computer Vision}, + pages={527--544}, + year={2016}, + organization={Springer} +} + +@inproceedings{goo2019one, + title={One-shot learning of multi-step tasks from observation via activity localization in auxiliary video}, + author={Goo, Wonjoon and Niekum, Scott}, + booktitle={2019 International Conference on Robotics and Automation (ICRA)}, + pages={7755--7761}, + year={2019}, + organization={IEEE} +} + +@inproceedings{dwibedi2018learning, + title={Learning actionable representations from visual observations}, + author={Dwibedi, Debidatta and Tompson, Jonathan and Lynch, Corey and Sermanet, Pierre}, + booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, + pages={1577--1584}, + year={2018}, + organization={IEEE} +} + +@inproceedings{sermanet2018time, + title={Time-contrastive networks: Self-supervised learning from video}, + author={Sermanet, Pierre and Lynch, Corey and Chebotar, Yevgen and Hsu, Jasmine and Jang, Eric and Schaal, Stefan and Levine, Sergey and Brain, Google}, + booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={1134--1141}, + year={2018}, + organization={IEEE} +} + +@article{kimura2018internal, + title={Internal model from observations for reward shaping}, + author={Kimura, Daiki and Chaudhury, Subhajit and Tachibana, Ryuki and Dasgupta, Sakyasingha}, + journal={arXiv preprint arXiv:1806.01267}, + year={2018} +} + +@article{espeholt2019seed, + title={SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference}, + author={Espeholt, Lasse and Marinier, Rapha{\"e}l and Stanczyk, Piotr and Wang, Ke and Michalski, Marcin}, + journal={arXiv preprint arXiv:1910.06591}, + year={2019} +} + + +@article{espeholt2018impala, + title={Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures}, + author={Espeholt, Lasse and Soyer, Hubert and Munos, Remi and Simonyan, Karen and Mnih, Volodymir and Ward, Tom and Doron, Yotam and Firoiu, Vlad and Harley, Tim and Dunning, Iain and others}, + journal={arXiv preprint arXiv:1802.01561}, + year={2018} +} + + +@article{jang2016categorical, + title={Categorical reparameterization with gumbel-softmax}, + author={Jang, Eric and Gu, Shixiang and Poole, Ben}, + journal={arXiv preprint arXiv:1611.01144}, + year={2016} +} + + +@book{puterman2014markov, + title={Markov decision processes: Discrete stochastic dynamic programming}, + author={Puterman, Martin L}, + year={2014}, + publisher={John Wiley \& Sons} +} + +@article{guo2019hybrid, + title={Hybrid reinforcement learning with expert state sequences}, + author={Guo, Xiaoxiao and Chang, Shiyu and Yu, Mo and Tesauro, Gerald and Campbell, Murray}, + journal={arXiv preprint arXiv:1903.04110}, + year={2019} +} + +@article{torabi2018behavioral, + title={Behavioral cloning from observation}, + author={Torabi, Faraz and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1805.01954}, + year={2018} +} + +@inproceedings{pathak2018zero, + title={Zero-shot visual imitation}, + author={Pathak, Deepak and Mahmoudieh, Parsa and Luo, Guanghao and Agrawal, Pulkit and Chen, Dian and Shentu, Yide and Shelhamer, Evan and Malik, Jitendra and Efros, Alexei A and Darrell, Trevor}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops}, + pages={2050--2053}, + year={2018} +} + +@article{pavse2019ridm, + title={RIDM: Reinforced Inverse Dynamics Modeling for Learning from a Single Observed Demonstration}, + author={Pavse, Brahma S and Torabi, Faraz and Hanna, Josiah P and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1906.07372}, + year={2019} +} + +@inproceedings{nair2017combining, + title={Combining self-supervised learning and imitation for vision-based rope manipulation}, + author={Nair, Ashvin and Chen, Dian and Agrawal, Pulkit and Isola, Phillip and Abbeel, Pieter and Malik, Jitendra and Levine, Sergey}, + booktitle={2017 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={2146--2153}, + year={2017}, + organization={IEEE} +} + +@article{edwards2018imitating, + title={Imitating latent policies from observation}, + author={Edwards, Ashley D and Sahni, Himanshu and Schroecker, Yannick and Isbell, Charles L}, + journal={arXiv preprint arXiv:1805.07914}, + year={2018} +} + +@inproceedings{hanna2017grounded, + title={Grounded action transformation for robot learning in simulation}, + author={Hanna, Josiah P and Stone, Peter}, + booktitle={Thirty-First AAAI Conference on Artificial Intelligence}, + year={2017} +} + +@article{stadie2017third, + title={Third-person imitation learning}, + author={Stadie, Bradly C and Abbeel, Pieter and Sutskever, Ilya}, + journal={arXiv preprint arXiv:1703.01703}, + year={2017} +} +@article{eysenbach2018diversity, + title={Diversity is all you need: Learning skills without a reward function}, + author={Eysenbach, Benjamin and Gupta, Abhishek and Ibarz, Julian and Levine, Sergey}, + journal={arXiv preprint arXiv:1802.06070}, + year={2018} +} + +@article{merel2017learning, + title={Learning human behaviors from motion capture by adversarial imitation}, + author={Merel, Josh and Tassa, Yuval and Srinivasan, Sriram and Lemmon, Jay and Wang, Ziyu and Wayne, Greg and Heess, Nicolas}, + journal={arXiv preprint arXiv:1707.02201}, + year={2017} +} + +@article{torabi2018behavioral, + title={Behavioral cloning from observation}, + author={Torabi, Faraz and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1805.01954}, + year={2018} +} + +@article{stadie2017third, + title={Third-person imitation learning}, + author={Stadie, Bradly C and Abbeel, Pieter and Sutskever, Ilya}, + journal={arXiv preprint arXiv:1703.01703}, + year={2017} +} + +@inproceedings{liu2018imitation, + title={Imitation from observation: Learning to imitate behaviors from raw video via context translation}, + author={Liu, YuXuan and Gupta, Abhishek and Abbeel, Pieter and Levine, Sergey}, + booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={1118--1125}, + year={2018}, + organization={IEEE} +} + +@article{gupta2017learning, + title={Learning invariant feature spaces to transfer skills with reinforcement learning}, + author={Gupta, Abhishek and Devin, Coline and Liu, YuXuan and Abbeel, Pieter and Levine, Sergey}, + journal={arXiv preprint arXiv:1703.02949}, + year={2017} +} + +@inproceedings{sermanet2018time, + title={Time-contrastive networks: Self-supervised learning from video}, + author={Sermanet, Pierre and Lynch, Corey and Chebotar, Yevgen and Hsu, Jasmine and Jang, Eric and Schaal, Stefan and Levine, Sergey and Brain, Google}, + booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={1134--1141}, + year={2018}, + organization={IEEE} +} + +@article{ziebart2010modeling, + title={Modeling interaction via the principle of maximum causal entropy}, + author={Ziebart, Brian D and Bagnell, J Andrew and Dey, Anind K}, + year={2010}, + publisher={figshare} +} + + +@inproceedings{syed2008apprenticeship, + title={Apprenticeship learning using linear programming}, + author={Syed, Umar and Bowling, Michael and Schapire, Robert E}, + booktitle={Proceedings of the 25th international conference on Machine learning}, + pages={1032--1039}, + year={2008}, + organization={ACM} +} + + +@inproceedings{ng2000algorithms, + title={Algorithms for inverse reinforcement learning.}, + author={Ng, Andrew Y and Russell, Stuart J and others}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + volume={1}, + pages={2}, + year={2000} +} + + +@inproceedings{ross2011reduction, + title={A reduction of imitation learning and structured prediction to no-regret online learning}, + author={Ross, St{\'e}phane and Gordon, Geoffrey and Bagnell, Drew}, + booktitle={Proceedings of the fourteenth international conference on artificial intelligence and statistics}, + pages={627--635}, + year={2011} +} + +@inproceedings{finn2016guided, + title={Guided cost learning: Deep inverse optimal control via policy optimization}, + author={Finn, Chelsea and Levine, Sergey and Abbeel, Pieter}, + booktitle={International Conference on Machine Learning}, + pages={49--58}, + year={2016} +} + + +@inproceedings{abbeel2004apprenticeship, + title={Apprenticeship learning via inverse reinforcement learning}, + author={Abbeel, Pieter and Ng, Andrew Y}, + booktitle={Proceedings of the twenty-first international conference on Machine learning}, + pages={1}, + year={2004}, + organization={ACM} +} + +@inproceedings{russell1998learning, + title={Learning agents for uncertain environments}, + author={Russell, Stuart J}, + booktitle={COLT}, + volume={98}, + pages={101--103}, + year={1998} +} + +@inproceedings{ross2010efficient, + title={Efficient reductions for imitation learning}, + author={Ross, St{\'e}phane and Bagnell, Drew}, + booktitle={Proceedings of the thirteenth international conference on artificial intelligence and statistics}, + pages={661--668}, + year={2010} +} + +@inproceedings{ross2011reduction, + title={A reduction of imitation learning and structured prediction to no-regret online learning}, + author={Ross, St{\'e}phane and Gordon, Geoffrey and Bagnell, Drew}, + booktitle={Proceedings of the fourteenth international conference on artificial intelligence and statistics}, + pages={627--635}, + year={2011} +} + + +@inproceedings{blau2018improving, + title={Improving Reinforcement Learning Pre-Training with Variational Dropout}, + author={Blau, Tom and Ott, Lionel and Ramos, Fabio}, + booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, + pages={4115--4122}, + year={2018}, + organization={IEEE} +} + +@inproceedings{berkenkamp2017safe, + title={Safe model-based reinforcement learning with stability guarantees}, + author={Berkenkamp, Felix and Turchetta, Matteo and Schoellig, Angela and Krause, Andreas}, + booktitle={Advances in Neural Information Processing Systems}, + pages={908--918}, + year={2017} +} + +@article{luo2019learning, + title={Learning self-correctable policies and value functions from demonstrations with negative sampling}, + author={Luo, Yuping and Xu, Huazhe and Ma, Tengyu}, + journal={arXiv preprint arXiv:1907.05634}, + year={2019} +} + +@article{garcia2015comprehensive, + title={A comprehensive survey on safe reinforcement learning}, + author={Garc{\i}a, Javier and Fern{\'a}ndez, Fernando}, + journal={Journal of Machine Learning Research}, + volume={16}, + number={1}, + pages={1437--1480}, + year={2015} +} + + +@inproceedings{chow2014algorithms, + title={Algorithms for CVaR optimization in MDPs}, + author={Chow, Yinlam and Ghavamzadeh, Mohammad}, + booktitle={Advances in Neural Information Processing Systems}, + pages={3509--3517}, + year={2014} +} + + +@article{finn2017one, + title={One-shot visual imitation learning via meta-learning}, + author={Finn, Chelsea and Yu, Tianhe and Zhang, Tianhao and Abbeel, Pieter and Levine, Sergey}, + journal={arXiv preprint arXiv:1709.04905}, + year={2017} +} + +@article{madumal2019explainable, + title={Explainable reinforcement learning through a causal lens}, + author={Madumal, Prashan and Miller, Tim and Sonenberg, Liz and Vetere, Frank}, + journal={arXiv preprint arXiv:1905.10958}, + year={2019} +} + +@inproceedings{duan2017one, + title={One-shot imitation learning}, + author={Duan, Yan and Andrychowicz, Marcin and Stadie, Bradly and Ho, OpenAI Jonathan and Schneider, Jonas and Sutskever, Ilya and Abbeel, Pieter and Zaremba, Wojciech}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1087--1098}, + year={2017} +} + + +@article{arulkumaran2017deep, + title={Deep reinforcement learning: A brief survey}, + author={Arulkumaran, Kai and Deisenroth, Marc Peter and Brundage, Miles and Bharath, Anil Anthony}, + journal={IEEE Signal Processing Magazine}, + volume={34}, + number={6}, + pages={26--38}, + year={2017}, + publisher={IEEE} +} + + +@inproceedings{koutnik2013evolving, + title={Evolving large-scale neural networks for vision-based reinforcement learning}, + author={Koutn{\'\i}k, Jan and Cuccu, Giuseppe and Schmidhuber, J{\"u}rgen and Gomez, Faustino}, + booktitle={Proceedings of the 15th annual conference on Genetic and evolutionary computation}, + pages={1061--1068}, + year={2013}, + organization={ACM} +} + + +@article{bojarski2016end, + title={End to end learning for self-driving cars}, + author={Bojarski, Mariusz and Del Testa, Davide and Dworakowski, Daniel and Firner, Bernhard and Flepp, Beat and Goyal, Prasoon and Jackel, Lawrence D and Monfort, Mathew and Muller, Urs and Zhang, Jiakai and others}, + journal={arXiv preprint arXiv:1604.07316}, + year={2016} +} + + +@inproceedings{molchanov2017variational, + title={Variational dropout sparsifies deep neural networks}, + author={Molchanov, Dmitry and Ashukha, Arsenii and Vetrov, Dmitry}, + booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70}, + pages={2498--2507}, + year={2017}, + organization={JMLR. org} +} + + +@inproceedings{blau2018improving, + title={Improving Reinforcement Learning Pre-Training with Variational Dropout}, + author={Blau, Tom and Ott, Lionel and Ramos, Fabio}, + booktitle={2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, + pages={4115--4122}, + year={2018}, + organization={IEEE} +} + + +@inproceedings{pastor2009learning, + title={Learning and generalization of motor skills by learning from demonstration}, + author={Pastor, Peter and Hoffmann, Heiko and Asfour, Tamim and Schaal, Stefan}, + booktitle={2009 IEEE International Conference on Robotics and Automation}, + pages={763--768}, + year={2009}, + organization={IEEE} +} + +@inproceedings{papavassiliou1999convergence, + title={Convergence of reinforcement learning with general function approximators}, + author={Papavassiliou, Vassilis A and Russell, Stuart}, + booktitle={International Joint Conference on Artificial Intelligence}, + volume={99}, + pages={748--755}, + year={1999} +} + +@inproceedings{ng2000algorithms, + title={Algorithms for inverse reinforcement learning.}, + author={Ng, Andrew Y and Russell, Stuart J and others}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + volume={1}, + pages={2}, + year={2000} +} + + +@inproceedings{ziebart2008maximum, + title={Maximum entropy inverse reinforcement learning.}, + author={Ziebart, Brian D and Maas, Andrew L and Bagnell, J Andrew and Dey, Anind K}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + volume={8}, + pages={1433--1438}, + year={2008}, + organization={Chicago, IL, USA} +} + + +@inproceedings{russell1998learning, + title={Learning agents for uncertain environments}, + author={Russell, Stuart J}, + booktitle={COLT}, + volume={98}, + pages={101--103}, + year={1998} +} + + +@inproceedings{ng2000algorithms, + title={Algorithms for inverse reinforcement learning.}, + author={Ng, Andrew Y and Russell, Stuart J and others}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + volume={1}, + pages={2}, + year={2000} +} + +@inproceedings{ho2016generative, + title={Generative adversarial imitation learning}, + author={Ho, Jonathan and Ermon, Stefano}, + booktitle={Advances in Neural Information Processing Systems}, + pages={4565--4573}, + year={2016} +} + +@inproceedings{hausman2017multi, + title={Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets}, + author={Hausman, Karol and Chebotar, Yevgen and Schaal, Stefan and Sukhatme, Gaurav and Lim, Joseph J}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1235--1245}, + year={2017} +} + +@article{sutton1999between, + title={Between {MDPs} and {semi-MDPs}: A framework for temporal abstraction in reinforcement learning}, + author={Sutton, Richard S and Precup, Doina and Singh, Satinder}, + journal={Artificial intelligence}, + volume={112}, + number={1-2}, + pages={181--211}, + year={1999}, + publisher={Elsevier} +} + +@inproceedings{dayan1993feudal, + title={Feudal reinforcement learning}, + author={Dayan, Peter and Hinton, Geoffrey E}, + booktitle={Advances in Neural Information Processing Systems}, + pages={271--278}, + year={1993} +} + +@article{bhatti2016playing, + title={Playing {Doom} with slam-augmented deep reinforcement learning}, + author={Bhatti, Shehroze and Desmaison, Alban and Miksik, Ondrej and Nardelli, Nantas and Siddharth, N and Torr, Philip HS}, + journal={arXiv preprint arXiv:1612.00380}, + year={2016} +} + +@inproceedings{kempka2016vizdoom, + title={{ViZDoom}: A {Doom}-based {AI} research platform for visual reinforcement learning}, + author={Kempka, Micha{\l} and Wydmuch, Marek and Runc, Grzegorz and Toczek, Jakub and Ja{\'s}kowski, Wojciech}, + booktitle={2016 IEEE Conference on Computational Intelligence and Games (CIG)}, + pages={1--8}, + year={2016}, + organization={IEEE} +} + +@inproceedings{tamar2016value, + title={Value iteration networks}, + author={Tamar, Aviv and Wu, Yi and Thomas, Garrett and Levine, Sergey and Abbeel, Pieter}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2154--2162}, + year={2016} +} + +@article{levy2018hierarchical, + title={Hierarchical reinforcement learning with hindsight}, + author={Levy, Andrew and Platt, Robert and Saenko, Kate}, + journal={arXiv preprint arXiv:1805.08180}, + year={2018} +} + +@article{haarnoja2018latent, + title={Latent space policies for hierarchical reinforcement learning}, + author={Haarnoja, Tuomas and Hartikainen, Kristian and Abbeel, Pieter and Levine, Sergey}, + journal={arXiv preprint arXiv:1804.02808}, + year={2018} +} + +@inproceedings{mnih2014recurrent, + title={Recurrent models of visual attention}, + author={Mnih, Volodymyr and Heess, Nicolas and Graves, Alex and others}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2204--2212}, + year={2014} +} + +@phdthesis{schulman2016optimizing, + title={Optimizing expectations: From deep reinforcement learning to stochastic computation graphs}, + author={Schulman, John}, + year={2016}, + school={UC Berkeley} +} + +@article{sahni2017state, + title={State space decomposition and subgoal creation for transfer in deep reinforcement learning}, + author={Sahni, Himanshu and Kumar, Saurabh and Tejani, Farhan and Schroecker, Yannick and Isbell, Charles}, + journal={arXiv preprint arXiv:1705.08997}, + year={2017} +} + +@article{arulkumaran2016classifying, + title={Classifying options for deep reinforcement learning}, + author={Arulkumaran, Kai and Dilokthanakul, Nat and Shanahan, Murray and Bharath, Anil Anthony}, + journal={arXiv preprint arXiv:1604.08153}, + year={2016} +} + +@book{parr1998hierarchical, + title={Hierarchical control and learning for {Markov} decision processes}, + author={Parr, Ronald Edward and Russell, Stuart}, + year={1998}, + publisher={University of California, Berkeley Berkeley, CA} +} + +@article{dietterich2000hierarchical, + title={Hierarchical reinforcement learning with the {MAXQ} value function decomposition}, + author={Dietterich, Thomas G}, + journal={Journal of Artificial Intelligence Research}, + volume={13}, + pages={227--303}, + year={2000} +} + +@article{schulman2015high, + title={High-dimensional continuous control using generalized advantage estimation}, + author={Schulman, John and Moritz, Philipp and Levine, Sergey and Jordan, Michael and Abbeel, Pieter}, + journal={arXiv preprint arXiv:1506.02438}, + year={2015} +} + +@article{dhariwal2017openai, + title={{OpenAI} baselines}, + author={Dhariwal, Prafulla and Hesse, Christopher and Klimov, Oleg and Nichol, Alex and Plappert, Matthias and Radford, Alec and Schulman, John and Sidor, Szymon and Wu, Yuhuai and Zhokhov, Peter}, + journal={GitHub, GitHub repository}, + year={2017} +} + +@article{dayan1993improving, + title={Improving generalization for temporal difference learning: The successor representation}, + author={Dayan, Peter}, + journal={Neural Computation}, + volume={5}, + number={4}, + pages={613--624}, + year={1993}, + publisher={MIT Press} +} + +@inproceedings{kaelbling1993hierarchical, + title={Hierarchical learning in stochastic domains: Preliminary results}, + author={Kaelbling, Leslie Pack}, + booktitle={Proceedings of the tenth International Conference on Machine Learning (ICML)}, + volume={951}, + pages={167--173}, + year={1993} +} + +@inproceedings{parr1998reinforcement, + title={Reinforcement learning with hierarchies of machines}, + author={Parr, Ronald and Russell, Stuart J}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1043--1049}, + year={1998} +} + +@inproceedings{vezhnevets2016strategic, + title={Strategic attentive writer for learning macro-actions}, + author={Vezhnevets, Alexander and Mnih, Volodymyr and Osindero, Simon and Graves, Alex and Vinyals, Oriol and Agapiou, John and others}, + booktitle={Advances in Neural Information Processing Systems}, + pages={3486--3494}, + year={2016} +} + +@article{barto2003recent, + title={Recent advances in hierarchical reinforcement learning}, + author={Barto, Andrew G and Mahadevan, Sridhar}, + journal={Discrete event dynamic systems}, + volume={13}, + number={1-2}, + pages={41--77}, + year={2003}, + publisher={Springer} +} + +@inproceedings{bacon2017option, + title={The option-critic architecture}, + author={Bacon, Pierre-Luc and Harb, Jean and Precup, Doina}, + booktitle={Thirty-First AAAI Conference on Artificial Intelligence}, + year={2017} +} + +@inproceedings{konidaris2009skill, + title={Skill discovery in continuous reinforcement learning domains using skill chaining}, + author={Konidaris, George and Barto, Andrew G}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1015--1023}, + year={2009} +} + +@inproceedings{vezhnevets2017feudal, + title={Feudal networks for hierarchical reinforcement learning}, + author={Vezhnevets, Alexander Sasha and Osindero, Simon and Schaul, Tom and Heess, Nicolas and Jaderberg, Max and Silver, David and Kavukcuoglu, Koray}, + booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70}, + pages={3540--3549}, + year={2017}, + organization={JMLR. org} +} + +@inproceedings{dietterich1998maxq, + title={The {MAXQ} Method for Hierarchical Reinforcement Learning.}, + author={Dietterich, Thomas G}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + volume={98}, + pages={118--126}, + year={1998}, + organization={Citeseer} +} +@inproceedings{fox2016taming, + title={Taming the noise in reinforcement learning via soft updates}, + author={Fox, Roy and Pakman, Ari and Tishby, Naftali}, + booktitle={Proceedings of the Thirty-Second Conference on Uncertainty in Artificial Intelligence}, + pages={202--211}, + year={2016}, + organization={AUAI Press} +} +@inproceedings{nachum2018data, + title={Data-efficient hierarchical reinforcement learning}, + author={Nachum, Ofir and Gu, Shixiang Shane and Lee, Honglak and Levine, Sergey}, + booktitle={Advances in Neural Information Processing Systems}, + pages={3303--3313}, + year={2018} +} + +@article{nachum2019does, + title={Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?}, + author={Nachum, Ofir and Tang, Haoran and Lu, Xingyu and Gu, Shixiang and Lee, Honglak and Levine, Sergey}, + journal={arXiv preprint arXiv:1909.10618}, + year={2019} +} + +@phdthesis{precup2000temporal, + title={Temporal abstraction in reinforcement learning}, + author={Hausknecht, Matthew John}, + institution={University of Massachusetts, Amherst}, + year={2000} +} + +@article{wiering1997hq, + title={{HQ-learning}}, + author={Wiering, Marco and Schmidhuber, J{\"u}rgen}, + journal={Adaptive Behavior}, + volume={6}, + number={2}, + pages={219--246}, + year={1997}, + publisher={Sage Publications Sage CA: Thousand Oaks, CA} +} + +@inproceedings{schaul2015universal, + title={Universal value function approximators}, + author={Schaul, Tom and Horgan, Daniel and Gregor, Karol and Silver, David}, + booktitle={International Conference on Machine Learning}, + pages={1312--1320}, + year={2015} +} + +@inproceedings{tessler2017deep, + title={A deep hierarchical approach to lifelong learning in {Minecraft}}, + author={Tessler, Chen and Givony, Shahar and Zahavy, Tom and Mankowitz, Daniel J and Mannor, Shie}, + booktitle={Thirty-First AAAI Conference on Artificial Intelligence}, + year={2017} +} + +@inproceedings{kulkarni2016hierarchical, + title={Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation}, + author={Kulkarni, Tejas D and Narasimhan, Karthik and Saeedi, Ardavan and Tenenbaum, Josh}, + booktitle={Advances in Neural Information Processing Systems}, + pages={3675--3683}, + year={2016} +} + +@article{da2012learning, + title={Learning parameterized skills}, + author={Da Silva, Bruno and Konidaris, George and Barto, Andrew}, + journal={arXiv preprint arXiv:1206.6398}, + year={2012} +} + +@inproceedings{hauskrecht1998hierarchical, + title={Hierarchical solution of {Markov} decision processes using macro-actions}, + author={Hauskrecht, Milos and Meuleau, Nicolas and Kaelbling, Leslie Pack and Dean, Thomas and Boutilier, Craig}, + booktitle={Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence}, + pages={220--229}, + year={1998}, + organization={Morgan Kaufmann Publishers Inc.} +} + +@inproceedings{riemer2018learning, + title={Learning abstract options}, + author={Riemer, Matthew and Liu, Miao and Tesauro, Gerald}, + booktitle={Advances in Neural Information Processing Systems}, + pages={10424--10434}, + year={2018} +} + +@article{gregor2015draw, + title={Draw: A recurrent neural network for image generation}, + author={Gregor, Karol and Danihelka, Ivo and Graves, Alex and Rezende, Danilo Jimenez and Wierstra, Daan}, + journal={arXiv preprint arXiv:1502.04623}, + year={2015} +} + +@article{liu2017action, + title={Action-depedent Control Variates for Policy Optimization via Stein's Identity}, + author={Liu, Hao and Feng, Yihao and Mao, Yi and Zhou, Dengyong and Peng, Jian and Liu, Qiang}, + journal={arXiv preprint arXiv:1710.11198}, + year={2017} +} + +@article{li2018policy, + title={Policy optimization with second-order advantage information}, + author={Li, Jiajin and Wang, Baoxiang}, + journal={arXiv preprint arXiv:1805.03586}, + year={2018} +} + +@inproceedings{konda2000actor, + title={Actor-critic algorithms}, + author={Konda, Vijay R and Tsitsiklis, John N}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1008--1014}, + year={2000} +} + +@article{wu2018variance, + title={Variance reduction for policy gradient with action-dependent factorized baselines}, + author={Wu, Cathy and Rajeswaran, Aravind and Duan, Yan and Kumar, Vikash and Bayen, Alexandre M and Kakade, Sham and Mordatch, Igor and Abbeel, Pieter}, + journal={arXiv preprint arXiv:1803.07246}, + year={2018} +} + +@article{yu2015multi, + title={Multi-scale context aggregation by dilated convolutions}, + author={Yu, Fisher and Koltun, Vladlen}, + journal={arXiv preprint arXiv:1511.07122}, + year={2015} +} + +@article{florensa2017stochastic, + title={Stochastic neural networks for hierarchical reinforcement learning}, + author={Florensa, Carlos and Duan, Yan and Abbeel, Pieter}, + journal={arXiv preprint arXiv:1704.03012}, + year={2017} +} + +@article{sharma2017learning, + title={Learning to repeat: Fine grained action repetition for deep reinforcement learning}, + author={Sharma, Sahil and Lakshminarayanan, Aravind S and Ravindran, Balaraman}, + journal={arXiv preprint arXiv:1702.06054}, + year={2017} +} + +@inproceedings{machado2017laplacian, + title={A {Laplacian} framework for option discovery in reinforcement learning}, + author={Machado, Marios C and Bellemare, Marc G and Bowling, Michael}, + booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70}, + pages={2295--2304}, + year={2017}, + organization={JMLR. org} +} + +@inproceedings{harutyunyan2018learning, + title={Learning with options that terminate off-policy}, + author={Harutyunyan, Anna and Vrancx, Peter and Bacon, Pierre-Luc and Precup, Doina and Nowe, Ann}, + booktitle={Thirty-Second AAAI Conference on Artificial Intelligence}, + year={2018} +} + +@article{frans2017meta, + title={Meta learning shared hierarchies}, + author={Frans, Kevin and Ho, Jonathan and Chen, Xi and Abbeel, Pieter and Schulman, John}, + journal={arXiv preprint arXiv:1710.09767}, + year={2017} +} + +@article{levy2018learning, + title={Learning multi-level hierarchies with hindsight}, + author={Levy, Andrew and Konidaris, George and Platt, Robert and Saenko, Kate}, + year={2018} +} + +@techreport{krizhevsky2009learning, + title={Learning multiple layers of features from tiny images}, + author={Krizhevsky, Alex and Hinton, Geoffrey and others}, + year={2009}, + institution={Citeseer} +} + +% Arena + +@article{fairbairn1995go, + title={Go in ancient China}, + author={Fairbairn, John}, + journal={Go Base}, + year={1995} +} + +@article{espeholt2018impala, + title={Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures}, + author={Espeholt, Lasse and Soyer, Hubert and Munos, Remi and Simonyan, Karen and Mnih, Volodymir and Ward, Tom and Doron, Yotam and Firoiu, Vlad and Harley, Tim and Dunning, Iain and others}, + journal={arXiv:1802.01561}, + year={2018} +} + +@article{yan2018chalet, + title={CHALET: Cornell house agent learning environment}, + author={Yan, Claudia and Misra, Dipendra and Bennnett, Andrew and Walsman, Aaron and Bisk, Yonatan and Artzi, Yoav}, + journal={arXiv:1801.07357}, + year={2018} +} + +@article{savva2019habitat, + title={Habitat: A Platform for Embodied AI Research}, + author={Savva, Manolis and Kadian, Abhishek and Maksymets, Oleksandr and Zhao, Yili and Wijmans, Erik and Jain, Bhavana and Straub, Julian and Liu, Jia and Koltun, Vladlen and Malik, Jitendra and others}, + journal={arXiv:1904.01201}, + year={2019} +} + +@article{chang2017matterport3d, + title={Matterport3d: Learning from rgb-d data in indoor environments}, + author={Chang, Angel and Dai, Angela and Funkhouser, Thomas and Halber, Maciej and Niessner, Matthias and Savva, Manolis and Song, Shuran and Zeng, Andy and Zhang, Yinda}, + journal={arXiv:1709.06158}, + year={2017} +} + +@article{savva2017minos, + title={MINOS: Multimodal indoor simulator for navigation in complex environments}, + author={Savva, Manolis and Chang, Angel X and Dosovitskiy, Alexey and Funkhouser, Thomas and Koltun, Vladlen}, + journal={arXiv:1712.03931}, + year={2017} +} + +@article{hernandez2018multiagent, + title={Is multiagent deep reinforcement learning the answer or the question? A brief survey}, + author={Hernandez-Leal, Pablo and Kartal, Bilal and Taylor, Matthew E}, + journal={arXiv preprint arXiv:1810.05587}, + year={2018} +} + +@inproceedings{qiu2017unrealcv, + title={Unrealcv: Virtual worlds for computer vision}, + author={Qiu, Weichao and Zhong, Fangwei and Zhang, Yi and Qiao, Siyuan and Xiao, Zihao and Kim, Tae Soo and Wang, Yizhou}, + booktitle={Proceedings of the 25th ACM international conference on Multimedia}, + pages={1221--1224}, + year={2017}, + organization={ACM} +} + +@article{wu2018building, + title={Building generalizable agents with a realistic and rich 3D environment}, + author={Wu, Yi and Wu, Yuxin and Gkioxari, Georgia and Tian, Yuandong}, + journal={arXiv:1801.02209}, + year={2018} +} + +@article{brodeur2017home, + title={HoME: A household multimodal environment}, + author={Brodeur, Simon and Perez, Ethan and Anand, Ankesh and Golemo, Florian and Celotti, Luca and Strub, Florian and Rouat, Jean and Larochelle, Hugo and Courville, Aaron}, + journal={arXiv:1711.11017}, + year={2017} +} + +@article{kolve2017ai2, + title={Ai2-thor: An interactive 3d environment for visual ai}, + author={Kolve, Eric and Mottaghi, Roozbeh and Gordon, Daniel and Zhu, Yuke and Gupta, Abhinav and Farhadi, Ali}, + journal={arXiv:1712.05474}, + year={2017} +} + +@inproceedings{johnson2016malmo, + title={The Malmo Platform for Artificial Intelligence Experimentation.}, + author={Johnson, Matthew and Hofmann, Katja and Hutton, Tim and Bignell, David}, + booktitle={International Joint Conference on Artificial Intelligence}, + year={2016} +} + +@incollection{kidzinski2018learning, + title={Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments}, + author={Kidzi{\'n}ski, {\L}ukasz and Mohanty, Sharada Prasanna and Ong, Carmichael F and Huang, Zhewei and Zhou, Shuchang and Pechenko, Anton and Stelmaszczyk, Adam and Jarosik, Piotr and Pavlov, Mikhail and Kolesnikov, Sergey and others}, + booktitle={Advances in Neural Information Processing Systems Competition: Building Intelligent Systems}, + year={2018} +} + +@article{wydmuch2018vizdoom, + title={{ViZDoom} Competitions: Playing {Doom} from Pixels}, + author={Wydmuch, Marek and Kempka, Micha{\l} and Ja{\'s}kowski, Wojciech}, + journal={IEEE Transactions on Games}, + year={2018}, + publisher={IEEE} +} + +@article{leibo2018psychlab, + title={Psychlab: a psychology laboratory for deep reinforcement learning agents}, + author={Leibo, Joel Z and d'Autume, Cyprien de Masson and Zoran, Daniel and Amos, David and Beattie, Charles and Anderson, Keith and Casta{\~n}eda, Antonio Garc{\'\i}a and Sanchez, Manuel and Green, Simon and Gruslys, Audrunas and others}, + journal={arXiv:1801.08116}, + year={2018} +} + +@article{nichol2018gotta, + title={Gotta learn fast: A new benchmark for generalization in rl}, + author={Nichol, Alex and Pfau, Vicki and Hesse, Christopher and Klimov, Oleg and Schulman, John}, + journal={arXiv:1804.03720}, + year={2018} +} + +@article{balduzzi2019open, + title={Open-ended Learning in Symmetric Zero-sum Games}, + author={Balduzzi, David and Garnelo, Marta and Bachrach, Yoram and Czarnecki, Wojciech M and Perolat, Julien and Jaderberg, Max and Graepel, Thore}, + journal={arXiv:1901.08106}, + year={2019} +} + +@article{beattie2016deepmind, + title={{DeepMind Lab}}, + author={Beattie, Charles and Leibo, Joel Z and Teplyashin, Denis and Ward, Tom and Wainwright, Marcus and K{\"u}ttler, Heinrich and Lefrancq, Andrew and Green, Simon and Vald{\'e}s, V{\'\i}ctor and Sadik, Amir and others}, + journal={arXiv:1612.03801}, + year={2016} +} + +@inproceedings{tan1993multi, + title={Multi-agent reinforcement learning: Independent vs. cooperative agents}, + author={Tan, Ming}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={1993} +} + +@article{matignon2012independent, + title={Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems}, + author={Matignon, Laetitia and Laurent, Guillaume J and Le Fort-Piat, Nadine}, + journal={The Knowledge Engineering Review}, + year={2012} +} + +@article{stanley2004competitive, + title={Competitive coevolution through evolutionary complexification}, + author={Stanley, Kenneth O and Miikkulainen, Risto}, + journal={Journal of artificial intelligence research}, + year={2004} +} + +@article{cao2013overview, + title={An overview of recent progress in the study of distributed multi-agent coordination}, + author={Cao, Yongcan and Yu, Wenwu and Ren, Wei and Chen, Guanrong}, + journal={IEEE Transactions on Industrial informatics}, + year={2013} +} + +@article{ye2015multi, + title={A multi-agent framework for packet routing in wireless sensor networks}, + author={Ye, Dayong and Zhang, Minjie and Yang, Yun}, + journal={Sensors}, + year={2015} +} + +@phdthesis{hausknecht2016cooperation, + title={Cooperation and communication in multiagent deep reinforcement learning}, + author={Hausknecht, Matthew John}, + institution={The University of Texas at Austin}, + year={2016} +} + +@article{tuyls2012multiagent, + title={Multiagent learning: Basics, challenges, and prospects}, + author={Tuyls, Karl and Weiss, Gerhard}, + journal={Ai Magazine}, + volume={33}, + number={3}, + pages={41--41}, + year={2012} +} + + +@article{shoham2007if, + title={If multi-agent learning is the answer, what is the question?}, + author={Shoham, Yoav and Powers, Rob and Grenager, Trond}, + journal={Artificial Intelligence}, + year={2007} +} + +@inproceedings{schvartzman2009stronger, + title={Stronger CDA strategies through empirical game-theoretic analysis and reinforcement learning}, + author={Schvartzman, L Julian and Wellman, Michael P}, + booktitle={AAMAS}, + year={2009} +} + +@article{kouvaris2017evolution, + title={How evolution learns to generalise: Using the principles of learning theory to understand the evolution of developmental organisation}, + author={Kouvaris, Kostas and Clune, Jeff and Kounios, Loizos and Brede, Markus and Watson, Richard A}, + journal={PLoS}, + year={2017} +} + +@article{samothrakis2013coevolving, + title={Coevolving game-playing agents: Measuring performance and intransitivities}, + author={Samothrakis, Spyridon and Lucas, Simon and Runarsson, ThomasPhilip and Robles, David}, + journal={IEEE Transactions on Evolutionary Computation}, + year={2013} +} + + +@inproceedings{oliehoek2006parallel, + title={The parallel Nash memory for asymmetric games}, + author={Oliehoek, Frans A and De Jong, Edwin D and Vlassis, Nikos}, + booktitle={Proceedings of the 8th annual conference on Genetic and evolutionary computation}, + year={2006} +} + +@article{camerer2004cognitive, + title={A cognitive hierarchy model of games}, + author={Camerer, Colin F and Ho, Teck-Hua and Chong, Juin-Kuan}, + journal={The Quarterly Journal of Economics}, + year={2004} +} + +@article{costa2006cognition, + title={Cognition and behavior in two-person guessing games: An experimental study}, + author={Costa-Gomes, Miguel A and Crawford, Vincent P}, + journal={American Economic Review}, + year={2006} +} + +@article{wright2017predicting, + title={Predicting human behavior in unrepeated, simultaneous-move games}, + author={Wright, James R and Leyton-Brown, Kevin}, + journal={Games and Economic Behavior}, + year={2017} +} + +@inproceedings{wright2010beyond, + title={Beyond equilibrium: Predicting human behavior in normal-form games}, + author={Wright, James R and Leyton-Brown, Kevin}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2010} +} + +@inproceedings{wright2016using, + title={Using reinforcement learning to validate empirical game-theoretic analysis: A continuous double auction study}, + author={Wright, Mason}, + booktitle={CoRR}, + year={2016} +} + +@article{hofbauer2002global, + title={On the global convergence of stochastic fictitious play}, + author={Hofbauer, Josef and Sandholm, William H}, + journal={Econometrica}, + year={2002} +} + +@article{ponsen2009evolutionary, + title={An evolutionary game-theoretic analysis of poker strategies}, + author={Ponsen, Marc and Tuyls, Karl and Kaisers, Michael and Ramon, Jan}, + journal={Entertainment Computing}, + year={2009} +} + +@inproceedings{wellman2006methods, + title={Methods for empirical game-theoretic analysis}, + author={Wellman, Michael P}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2006} +} + +@inproceedings{walsh2002analyzing, + title={Analyzing complex strategic interactions in multi-agent systems}, + author={Walsh, William E and Das, Rajarshi and Tesauro, Gerald and Kephart, Jeffrey O}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2002} +} + +@inproceedings{bosansky2013using, + title={Using Double-oracle Method and Serialized Alpha-Beta Search for Pruning in Simultaneous Move Games}, + author={Bosansky, Branislav and Lisy, Viliam and Cermak, Jiri and Vitek, Roman and Pechoucek, Michal}, + booktitle={International Joint Conference on Artificial Intelligence}, + year={2013} +} + + +@inproceedings{zinkevich2007new, + title={A new algorithm for generating equilibria in massive zero-sum games}, + author={Zinkevich, Martin and Bowling, Michael and Burch, Neil}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2007} +} + + +@article{bovsansky2016algorithms, + title={Algorithms for computing strategies in two-player simultaneous move games}, + author={Bo{\v{s}}ansk{\`y}, Branislav and Lis{\`y}, Viliam and Lanctot, Marc and {\v{C}}erm{\'a}k, Ji{\v{r}}{\'\i} and Winands, Mark HM}, + journal={Artificial Intelligence}, + year={2016} +} + + +@inproceedings{mcmahan2003planning, + title={Planning in the presence of cost functions controlled by an adversary}, + author={McMahan, H Brendan and Gordon, Geoffrey J and Blum, Avrim}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2003} +} + +@inproceedings{lanctot2017unified, + title={A unified game-theoretic approach to multiagent reinforcement learning}, + author={Lanctot, Marc and Zambaldi, Vinicius and Gruslys, Audrunas and Lazaridou, Angeliki and Tuyls, Karl and P{\'e}rolat, Julien and Silver, David and Graepel, Thore}, + booktitle={Advances in Neural Information Processing Systems}, + pages={4190--4203}, + year={2017} +} + +@article{tampuu2017multiagent, + title={Multiagent cooperation and competition with deep reinforcement learning}, + author={Tampuu, Ardi and Matiisen, Tambet and Kodelja, Dorian and Kuzovkin, Ilya and Korjus, Kristjan and Aru, Juhan and Aru, Jaan and Vicente, Raul}, + journal={PloS one}, + year={2017} +} + +@article{heinrich2016deep, + title={Deep reinforcement learning from self-play in imperfect-information games}, + author={Heinrich, Johannes and Silver, David}, + journal={arXiv:1603.01121}, + year={2016} +} + +@inproceedings{das2017learning, + title={Learning cooperative visual dialog agents with deep reinforcement learning}, + author={Das, Abhishek and Kottur, Satwik and Moura, Jos{\'e} MF and Lee, Stefan and Batra, Dhruv}, + booktitle={ICCV}, + year={2017} +} + +@article{lerer2016learning, + title={Learning physical intuition of block towers by example}, + author={Lerer, Adam and Gross, Sam and Fergus, Rob}, + journal={arXiv:1603.01312}, + year={2016} +} + +@inproceedings{tian2017elf, + title={Elf: An extensive, lightweight and flexible research platform for real-time strategy games}, + author={Tian, Yuandong and Gong, Qucheng and Shang, Wenling and Wu, Yuxin and Zitnick, C Lawrence}, + booktitle={Advances in Neural Information Processing Systems}, + year={2017} +} + +@inproceedings{koller1999computing, + title={Computing factored value functions for policies in structured MDPs}, + author={Koller, Daphne and Parr, Ronald}, + booktitle={International Joint Conference on Artificial Intelligence}, + year={1999} +} + +@book{harris1998handbook, + title={Handbook of mathematics and computational science}, + author={Harris, John W and St{\"o}cker, Horst}, + year={1998}, + publisher={Springer Science \& Business Media} +} + +@article{leibo2018malthusian, + title={Malthusian Reinforcement Learning}, + author={Leibo, Joel Z and Perolat, Julien and Hughes, Edward and Wheelwright, Steven and Marblestone, Adam H and Du{\'e}{\~n}ez-Guzm{\'a}n, Edgar and Sunehag, Peter and Dunning, Iain and Graepel, Thore}, + journal={arXiv:1812.07019}, + year={2018} +} + +@inproceedings{cai2011minmax, + title={On minmax theorems for multiplayer games}, + author={Cai, Yang and Daskalakis, Constantinos}, + booktitle={Proceedings of the twenty-second annual ACM-SIAM symposium on Discrete Algorithms}, + year={2011}, + organization={Society for Industrial and Applied Mathematics} +} + +@article{rashid2018qmix, + title={QMIX: Monotonic value function factorisation for deep multi-agent reinforcement learning}, + author={Rashid, Tabish and Samvelyan, Mikayel and de Witt, Christian Schroeder and Farquhar, Gregory and Foerster, Jakob and Whiteson, Shimon}, + journal={arXiv:1803.11485}, + year={2018} +} + +@inproceedings{sunehag2018value, + title={Value-decomposition networks for cooperative multi-agent learning based on team reward}, + author={Sunehag, Peter and Lever, Guy and Gruslys, Audrunas and Czarnecki, Wojciech Marian and Zambaldi, Vinicius and Jaderberg, Max and Lanctot, Marc and Sonnerat, Nicolas and Leibo, Joel Z and Tuyls, Karl and others}, + booktitle={AAMAS}, + year={2018} +} + +@inproceedings{guestrin2002multiagent, + title={Multiagent planning with factored MDPs}, + author={Guestrin, Carlos and Koller, Daphne and Parr, Ronald}, + booktitle={Advances in Neural Information Processing Systems}, + year={2002} +} + +@inproceedings{lehman2008exploiting, + title={Exploiting open-endedness to solve problems through the search for novelty.}, + author={Lehman, Joel and Stanley, Kenneth O}, + booktitle={ALIFE}, + year={2008} +} + +@inproceedings{marecki2008not, + title={Not all agents are equal: scaling up distributed POMDPs for agent networks}, + author={Marecki, Janusz and Gupta, Tapana and Varakantham, Pradeep and Tambe, Milind and Yokoo, Makoto}, + booktitle={AAMAS}, + year={2008} +} + +@article{gmytrasiewicz2005framework, + title={A framework for sequential planning in multi-agent settings}, + author={Gmytrasiewicz, Piotr J and Doshi, Prashant}, + journal={Journal of Artificial Intelligence Research}, + year={2005} +} + +@inproceedings{amato2015scalable, + title={Scalable planning and learning for multiagent POMDPs}, + author={Amato, Christopher and Oliehoek, Frans A}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2015} +} + +@inproceedings{hoang2013interactive, + title={Interactive POMDP Lite: Towards practical planning to predict and exploit intentions for interacting with self-interested agents}, + author={Hoang, Trong Nghia and Low, Kian Hsiang}, + booktitle={International Joint Conference on Artificial Intelligence}, + year={2013} +} + +@book{nair2004coordinating, + title={Coordinating multiagent teams in uncertain domains using distributed POMDPs}, + author={Nair, Ranjit and Tambe, Milind}, + year={2004}, + publisher={University of Southern California} +} + +@inproceedings{omidshafiei2017deep, + title={Deep decentralized multi-task multi-agent reinforcement learning under partial observability}, + author={Omidshafiei, Shayegan and Pazis, Jason and Amato, Christopher and How, Jonathan P and Vian, John}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2017} +} + +@book{oliehoek2016concise, + title={A concise introduction to decentralized POMDPs}, + author={Oliehoek, Frans A and Amato, Christopher and others}, + year={2016} +} + + +@inproceedings{foerster2017stabilising, + title={Stabilising experience replay for deep multi-agent reinforcement learning}, + author={Foerster, Jakob and Nardelli, Nantas and Farquhar, Gregory and Afouras, Triantafyllos and Torr, Philip HS and Kohli, Pushmeet and Whiteson, Shimon}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2017} +} + +@inproceedings{gupta2017cooperative, + title={Cooperative multi-agent control using deep reinforcement learning}, + author={Gupta, Jayesh K and Egorov, Maxim and Kochenderfer, Mykel}, + booktitle={International Conference on Autonomous Agents and Multiagent Systems}, + pages={66--83}, + year={2017}, + organization={Springer} +} + +@inproceedings{heinrich2015fictitious, + title={Fictitious self-play in extensive-form games}, + author={Heinrich, Johannes and Lanctot, Marc and Silver, David}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2015} +} + +@inproceedings{yakovenko2016poker, + title={Poker-{CNN}: A pattern learning strategy for making draws and bets in poker games using convolutional networks}, + author={Yakovenko, Nikolai and Cao, Liangliang and Raffel, Colin and Fan, James}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2016} +} + +@inproceedings{pinto2017supervision, + title={Supervision via competition: Robot adversaries for learning tasks}, + author={Pinto, Lerrel and Davidson, James and Gupta, Abhinav}, + booktitle={International Conference on Robotics and Automation (ICRA)}, + year={2017} +} + +@inproceedings{sukhbaatar2016learning, + title={Learning multiagent communication with backpropagation}, + author={Sukhbaatar, Sainbayar and Fergus, Rob and others}, + booktitle={Advances in Neural Information Processing Systems}, + year={2016} +} + +@book{shoham2008multiagent, + title={Multiagent systems: Algorithmic, game-theoretic, and logical foundations}, + author={Shoham, Yoav and Leyton-Brown, Kevin}, + year={2008}, + publisher={Cambridge University Press} +} + +@article{peng2017multiagent, + title={Multiagent bidirectionally-coordinated nets for learning to play starcraft combat games}, + author={Peng, Peng and Yuan, Quan and Wen, Ying and Yang, Yaodong and Tang, Zhenkun and Long, Haitao and Wang, Jun}, + journal={arXiv:1703.10069}, + year={2017} +} + +@article{usunier2016episodic, + title={Episodic exploration for deep deterministic policies: An application to starcraft micromanagement tasks}, + author={Usunier, Nicolas and Synnaeve, Gabriel and Lin, Zeming and Chintala, Soumith}, + journal={arXiv:1609.02993}, + year={2016} +} + +@article{oliehoek2008optimal, + title={Optimal and approximate Q-value functions for decentralized POMDPs}, + author={Oliehoek, Frans A and Spaan, Matthijs TJ and Vlassis, Nikos}, + journal={Journal of Artificial Intelligence Research}, + year={2008} +} + +@inproceedings{chang2004all, + title={All learning is local: Multi-agent learning in global reward games}, + author={Chang, Yu-Han and Ho, Tracey and Kaelbling, Leslie P}, + booktitle={Advances in Neural Information Processing Systems}, + year={2004} +} + +@article{jorge2016learning, + title={Learning to play guess who? and inventing a grounded language as a consequence}, + author={Jorge, Emilio and K{\aa}geb{\"a}ck, Mikael and Johansson, Fredrik D and Gustavsson, Emil}, + journal={arXiv:1611.03218}, + year={2016} +} + +@article{kraemer2016multi, + title={Multi-agent reinforcement learning as a rehearsal for decentralized planning}, + author={Kraemer, Landon and Banerjee, Bikramjit}, + journal={Neurocomputing}, + year={2016} +} + +@inproceedings{hu1998online, + title={Online learning about other agents in a dynamic multiagent system}, + author={Hu, Junling and Wellman, Michael P}, + booktitle={AAMAS}, + year={1998} +} + +@inproceedings{boutilier1996learning, + title={Learning conventions in multiagent stochastic domains using likelihood estimates}, + author={Boutilier, Craig}, + booktitle={UAI}, + year={1996} +} + +@inproceedings{foerster2016learning, + title={Learning to communicate with deep multi-agent reinforcement learning}, + author={Foerster, Jakob and Assael, Ioannis Alexandros and de Freitas, Nando and Whiteson, Shimon}, + booktitle={Advances in Neural Information Processing Systems}, + year={2016} +} + +@inproceedings{mordatch2018emergence, + title={Emergence of grounded compositional language in multi-agent populations}, + author={Mordatch, Igor and Abbeel, Pieter}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2018} +} + +@inproceedings{chalkiadakis2003coordination, + title={Coordination in multiagent reinforcement learning: a Bayesian approach}, + author={Chalkiadakis, Georgios and Boutilier, Craig}, + booktitle={AAMAS}, + year={2003} +} + +@inproceedings{foerster2018counterfactual, + title={Counterfactual multi-agent policy gradients}, + author={Foerster, Jakob N and Farquhar, Gregory and Afouras, Triantafyllos and Nardelli, Nantas and Whiteson, Shimon}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2018} +} + + +@inproceedings{kapturowski2018recurrent, +title={Recurrent Experience Replay in Distributed Reinforcement Learning}, +author={Steven Kapturowski and Georg Ostrovski and Will Dabney and John Quan and Remi Munos}, +booktitle={International Conference on Learning Representations}, +year={2019}, +url={https://openreview.net/forum?id=r1lyTjAqYX}, +} + +@article{song2019v, + title={V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control}, + author={Song, H Francis and Abdolmaleki, Abbas and Springenberg, Jost Tobias and Clark, Aidan and Soyer, Hubert and Rae, Jack W and Noury, Seb and Ahuja, Arun and Liu, Siqi and Tirumala, Dhruva and others}, + journal={arXiv preprint arXiv:1909.12238}, + year={2019} +} + + +@article{abdolmaleki2018maximum, + title={Maximum a posteriori policy optimisation}, + author={Abdolmaleki, Abbas and Springenberg, Jost Tobias and Tassa, Yuval and Munos, Remi and Heess, Nicolas and Riedmiller, Martin}, + journal={arXiv preprint arXiv:1806.06920}, + year={2018} +} + + + +@article{heess2017emergence, + title={Emergence of locomotion behaviours in rich environments}, + author={Heess, Nicolas and Sriram, Srinivasan and Lemmon, Jay and Merel, Josh and Wayne, Greg and Tassa, Yuval and Erez, Tom and Wang, Ziyu and Eslami, SM and Riedmiller, Martin and others}, + journal={arXiv:1707.02286}, + year={2017} +} + +@inproceedings{lowe2017multi, + title={Multi-agent actor-critic for mixed cooperative-competitive environments}, + author={Lowe, Ryan and Wu, Yi and Tamar, Aviv and Harb, Jean and Abbeel, OpenAI Pieter and Mordatch, Igor}, + booktitle={Advances in Neural Information Processing Systems}, + year={2017} +} + +@article{lerer2017maintaining, + title={Maintaining cooperation in complex social dilemmas using deep reinforcement learning}, + author={Lerer, Adam and Peysakhovich, Alexander}, + journal={arXiv:1707.01068}, + year={2017} +} + +@inproceedings{foerster2018learning, + title={Learning with opponent-learning awareness}, + author={Foerster, Jakob and Chen, Richard Y and Al-Shedivat, Maruan and Whiteson, Shimon and Abbeel, Pieter and Mordatch, Igor}, + booktitle={AAMAS}, + year={2018} +} + +@book{luce1989games, + title={Games and decisions: Introduction and critical survey}, + author={Luce, R Duncan and Raiffa, Howard}, + year={1989} +} + +@inproceedings{he2016opponent, + title={Opponent modeling in deep reinforcement learning}, + author={He, He and Boyd-Graber, Jordan and Kwok, Kevin and Daum{\'e} III, Hal}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2016} +} + +@inproceedings{sukhbaatar2018intrinsic, + title={Intrinsic motivation and automatic curricula via asymmetric self-play}, + author={Sukhbaatar, Sainbayar and Lin, Zeming and Kostrikov, Ilya and Synnaeve, Gabriel and Szlam, Arthur and Fergus, Rob}, + booktitle={International Conference on Learning Representations}, + year={2018} +} + +@article{juliani2018unity, + title={Unity: A general platform for intelligent agents}, + author={Juliani, Arthur and Berges, Vincent-Pierre and Vckay, Esh and Gao, Yuan and Henry, Hunter and Mattar, Marwan and Lange, Danny}, + journal={arXiv:1809.02627}, + year={2018} +} + +@article{panait2005cooperative, + title={Cooperative multi-agent learning: The state of the art}, + author={Panait, Liviu and Luke, Sean}, + journal={Autonomous agents and multi-agent systems}, + year={2005} +} + +@article{bu2008comprehensive, + title={A comprehensive survey of multiagent reinforcement learning}, + author={Bu, Lucian and Babu, Robert and De Schutter, Bart and others}, + journal={IEEE Transactions on Systems, Man, and Cybernetics}, + year={2008} +} + +@incollection{samuel1988some, + title={Some Studies in Machine Learning Using the Game of Checkers.}, + author={Samuel, Arthur L}, + booktitle={Computer Games I}, + year={1988} +} + +@article{tesauro1995temporal, + title={Temporal difference learning and TD-Gammon}, + author={Tesauro, Gerald}, + journal={Communications of the ACM}, + year={1995} +} + +@article{leslie2006generalised, + title={Generalised weakened fictitious play}, + author={Leslie, David S and Collins, Edmund J}, + journal={Games and Economic Behavior}, + year={2006} +} + +@inproceedings{lanctot2009monte, + title={Monte Carlo sampling for regret minimization in extensive games}, + author={Lanctot, Marc and Waugh, Kevin and Zinkevich, Martin and Bowling, Michael}, + booktitle={Advances in Neural Information Processing Systems}, + year={2009} +} + + +@inproceedings{zinkevich2008regret, + title={Regret minimization in games with incomplete information}, + author={Zinkevich, Martin and Johanson, Michael and Bowling, Michael and Piccione, Carmelo}, + booktitle={Advances in Neural Information Processing Systems}, + year={2008} +} + + +@article{bosansky2014exact, + title={An exact double-oracle algorithm for zero-sum extensive-form games with imperfect information}, + author={Bosansky, Branislav and Kiekintveld, Christopher and Lisy, Viliam and Pechoucek, Michal}, + journal={Journal of Artificial Intelligence Research}, + year={2014} +} + +@article{miltersen2010computing, + title={Computing a quasi-perfect equilibrium of a two-player game}, + author={Miltersen, Peter Bro and S{\o}rensen, Troels Bjerre}, + journal={Economic Theory}, + year={2010} +} + +@inproceedings{gilpin2007gradient, + title={Gradient-based algorithms for finding Nash equilibria in extensive form games}, + author={Gilpin, Andrew and Hoda, Samid and Pena, Javier and Sandholm, Tuomas}, + booktitle={International Workshop on Web and Internet Economics}, + year={2007} +} + +@article{koller1996efficient, + title={Efficient computation of equilibria for extensive two-person games}, + author={Koller, Daphne and Megiddo, Nimrod and Von Stengel, Bernhard}, + journal={Games and economic behavior}, + volume={14}, + number={2}, + pages={247--259}, + year={1996}, + publisher={Elsevier} +} +} + +@article{bowling2015heads, + title={Heads-up limit hold’em poker is solved}, + author={Bowling, Michael and Burch, Neil and Johanson, Michael and Tammelin, Oskari}, + journal={Science}, + year={2015} +} + +@article{sandholm2010state, + title={The state of solving large incomplete-information games, and application to poker}, + author={Sandholm, Tuomas}, + journal={AI Magazine}, + year={2010} +} + +@article{campbell2002deep, + title={Deep blue}, + author={Campbell, Murray and Hoane Jr, A Joseph and Hsu, Feng-hsiung}, + journal={Artificial intelligence}, + year={2002} +} + +% @article{campbell2002deep, +% title={Deep blue}, +% author={Campbell, Murray and Hoane Jr, A Joseph and Hsu, Feng-hsiung}, +% journal={Artificial intelligence}, +% volume={134}, +% number={1-2}, +% pages={57--83}, +% year={2002}, +% publisher={Elsevier} +% } + +@inproceedings{heinrich2015smooth, + title={Smooth uct search in computer poker}, + author={Heinrich, Johannes and Silver, David}, + booktitle={International Joint Conference on Artificial Intelligence}, + year={2015} +} + +@inproceedings{ganzfried2009computing, + title={Computing equilibria in multiplayer stochastic games of imperfect information}, + author={Ganzfried, Sam and Sandholm, Tuomas}, + booktitle={International Joint Conference on Artificial Intelligence}, + year={2009} +} + +@book{puterman2014markov, + title={Markov decision processes: discrete stochastic dynamic programming}, + author={Puterman, Martin L}, + year={2014}, + publisher={John Wiley \& Sons} +} + +@book{myerson2013game, + title={Game theory}, + author={Myerson, Roger B}, + year={2013}, + publisher={Harvard university press} +} + + +@inproceedings{mcmahan2007fast, + title={A fast bundle-based anytime algorithm for poker and other convex games}, + author={McMahan, H Brendan and Gordony, Geoffrey J}, + booktitle={Artificial Intelligence and Statistics}, + year={2007} +} + +@article{lambert2005fictitious, + title={A fictitious play approach to large-scale optimization}, + author={Lambert Iii, Theodore J and Epelman, Marina A and Smith, Robert L}, + journal={Operations Research}, + year={2005} +} + +@misc{animal_ai, + title = {Animal-AI Olympics}, + author={Matthew, Crosby and Benjamin, Beyret and Murray, Shanahan and Marta, Halina and Lucy, Cheke}, + howpublished = {\url{http://animalaiolympics.com/}}, + year = {2019} +} + +@misc{OpenAI_dota, + author = {OpenAI}, + title = {OpenAI Five}, + howpublished = {\url{https://blog.openai.com/openai-five/}}, + year = {2018} +} + +@misc{AIIDE, + author = {David, Churchill}, + title = {AIIDE StarCraft AI Competitions}, + howpublished = {\url{https://www.cs.mun.ca/~dchurchill/starcraftaicomp/history.shtml}}, + year = {2016} +} + +@article{brown1951iterative, + title={Iterative solution of games by fictitious play}, + author={Brown, George}, + journal={Activity analysis of production and allocation}, + year={1951} +} + +@misc{OpenAI_Universe, + author = {OpenAI}, + title = {OpenAI Universe}, + howpublished = {\url{https://github.com/openai/universe}}, + year = {2016} +} + +@misc{AlphaStar, + author = {DeepMind}, + title = {AlphaStar: Mastering the Real-Time Strategy Game StarCraft II}, + howpublished = {\url{https://deepmind.com/blog/alphastar-mastering-real-time-strategy-game-starcraft-ii/}}, + year = {2019} +} + +@article{silver2016mastering, + title={Mastering the game of Go with deep neural networks and tree search}, + author={Silver, David and Huang, Aja and Maddison, Chris J and Guez, Arthur and Sifre, Laurent and Van Den Driessche, George and Schrittwieser, Julian and Antonoglou, Ioannis and Panneershelvam, Veda and Lanctot, Marc and others}, + journal={Nature}, + year={2016} +} + +@article{roberson2006colonel, + title={The colonel blotto game}, + author={Roberson, Brian}, + journal={Economic Theory}, + year={2006} +} + +@inproceedings{singh2009rewards, + title={Where do rewards come from}, + author={Singh, Satinder and Lewis, Richard L and Barto, Andrew G}, + booktitle={Proceedings of the annual conference of the cognitive science society}, + year={2009} +} + +@article{singh2010intrinsically, + title={Intrinsically motivated reinforcement learning: An evolutionary perspective}, + author={Singh, Satinder and Lewis, Richard L and Barto, Andrew G and Sorg, Jonathan}, + journal={IEEE Transactions on Autonomous Mental Development}, + year={2010} +} + +@article{rosin1997new, + title={New methods for competitive coevolution}, + author={Rosin, Christopher D and Belew, Richard K}, + journal={Evolutionary computation}, + year={1997} +} + +@incollection{hendtlass2004introduction, + title={An introduction to collective intelligence}, + author={Hendtlass, Tim}, + booktitle={Applied Intelligent Systems}, + year={2004} +} + +@inproceedings{jaderberg2018human, + title={Human-level performance in first-person multiplayer games with population-based deep reinforcement learning}, + author={Jaderberg, Max and Czarnecki, Wojciech M and Dunning, Iain and Marris, Luke and Lever, Guy and Castaneda, Antonio Garcia and Beattie, Charles and Rabinowitz, Neil C and Morcos, Ari S and Ruderman, Avraham and others}, + booktitle={CoRR}, + year={2018} +} + +@article{riedmiller2009reinforcement, + title={Reinforcement learning for robot soccer}, + author={Riedmiller, Martin and Gabel, Thomas and Hafner, Roland and Lange, Sascha}, + journal={Autonomous Robots}, + year={2009} +} + +@article{macalpine2018overlapping, + title={Overlapping layered learning}, + author={MacAlpine, Patrick and Stone, Peter}, + journal={Artificial Intelligence}, + year={2018} +} + +@inproceedings{todorov2012mujoco, + title={Mujoco: A physics engine for model-based control}, + author={Todorov, Emanuel and Erez, Tom and Tassa, Yuval}, + booktitle={IROS}, + year={2012} +} + +@article{tassa2018deepmind, + title={Deepmind control suite}, + author={Tassa, Yuval and Doron, Yotam and Muldal, Alistair and Erez, Tom and Li, Yazhe and Casas, Diego de Las and Budden, David and Abdolmaleki, Abbas and Merel, Josh and Lefrancq, Andrew and others}, + journal={arXiv:1801.00690}, + year={2018} +} + +@article{brockman2016openai, + title={{OpenAI} gym}, + author={Brockman, Greg and Cheung, Vicki and Pettersson, Ludwig and Schneider, Jonas and Schulman, John and Tang, Jie and Zaremba, Wojciech}, + journal={arXiv:1606.01540}, + year={2016} +} + +@article{bellemare2013arcade, + title={The {A}rcade {L}earning {E}nvironment: An evaluation platform for general agents}, + author={Bellemare, Marc G and Naddaf, Yavar and Veness, Joel and Bowling, Michael}, + journal={Journal of Artificial Intelligence Research}, + volume={47}, + pages={253--279}, + year={2013} +} + +@book{cooper1999coordination, + title={Coordination games}, + author={Cooper, Russell}, + year={1999}, + publisher={Cambridge University Press} +} + + +@book{rapoport1965prisoner, + title={Prisoner's dilemma: A study in conflict and cooperation}, + author={Rapoport, Anatol and Chammah, Albert M and Orwant, Carol J}, + volume={165}, + year={1965}, + publisher={University of Michigan press} +} + + +@inproceedings{riedmiller2018learning, + title={Learning by playing-solving sparse reward tasks from scratch}, + author={Riedmiller, Martin and Hafner, Roland and Lampe, Thomas and Neunert, Michael and Degrave, Jonas and Van de Wiele, Tom and Mnih, Volodymyr and Heess, Nicolas and Springenberg, Jost Tobias}, + booktitle={PMLR}, + year={2018} +} + + +@inproceedings{balduzzi2018re, + title={Re-evaluating Evaluation}, + author={Balduzzi, David and Tuyls, Karl and Perolat, Julien and Graepel, Thore}, + booktitle={Advances in Neural Information Processing Systems}, + year={2018} +} + +@article{schulman2017proximal, + title={Proximal policy optimization algorithms}, + author={Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg}, + journal={arXiv:1707.06347}, + year={2017} +} + +@inproceedings{tobin2017domain, + title={Domain randomization for transferring deep neural networks from simulation to the real world}, + author={Tobin, Josh and Fong, Rachel and Ray, Alex and Schneider, Jonas and Zaremba, Wojciech and Abbeel, Pieter}, + booktitle={ROS}, + year={2017} +} + + +@inproceedings{osawa1996robocup, + title={RoboCup: The robot world cup initiative}, + author={Osawa, Eiichi and Kitano, Hiroaki and Asada, Minoru and Kuniyoshi, Yasuo and Noda, Itsuki}, + booktitle={ICMAS}, + year={1996} +} + +@inproceedings{kleiman2016coordinate, + title={Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction}, + author={Kleiman-Weiner, Max and Ho, Mark K and Austerweil, Joseph L and Littman, Michael L and Tenenbaum, Joshua B}, + booktitle={CogSci}, + year={2016} +} + +@inproceedings{littman2001friend, + title={Friend-or-foe Q-learning in general-sum games}, + author={Littman, Michael L}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2001} +} + +@article{panait2008theoretical, + title={Theoretical advantages of lenient learners: An evolutionary game theoretic perspective}, + author={Panait, Liviu and Tuyls, Karl and Luke, Sean}, + journal={Journal of Machine Learning Research}, + year={2008} +} + +@inproceedings{kaisers2010frequency, + title={Frequency adjusted multi-agent Q-learning}, + author={Kaisers, Michael and Tuyls, Karl}, + booktitle={AAMAS}, + year={2010} +} + +@article{bowling2002multiagent, + title={Multiagent learning using a variable learning rate}, + author={Bowling, Michael and Veloso, Manuela}, + journal={Artificial Intelligence}, + year={2002} +} + +@inproceedings{lauer2004reinforcement, + title={Reinforcement learning for stochastic cooperative multi-agent systems}, + author={Lauer, Martin and Riedmiller, Martin}, + booktitle={AAMAS}, + year={2004} +} + + +@inproceedings{greenwald2003correlated, + title={Correlated Q-learning}, + author={Greenwald, Amy and Hall, Keith and Serrano, Roberto}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2003} +} + +@article{claus1998dynamics, + title={The dynamics of reinforcement learning in cooperative multiagent systems}, + author={Claus, Caroline and Boutilier, Craig}, + journal={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={1998} +} + + +@incollection{littman1994markov, + title={Markov games as a framework for multi-agent reinforcement learning}, + author={Littman, Michael L}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={1994} +} + +@article{moravvcik2017deepstack, + title={Deepstack: Expert-level artificial intelligence in heads-up no-limit poker}, + author={Morav{\v{c}}{\'\i}k, Matej and Schmid, Martin and Burch, Neil and Lis{\`y}, Viliam and Morrill, Dustin and Bard, Nolan and Davis, Trevor and Waugh, Kevin and Johanson, Michael and Bowling, Michael}, + journal={Science}, + year={2017} +} + +@inproceedings{williams2016ms, + title={Ms. pac-man versus ghost team CIG 2016 competition}, + author={Williams, Piers R and Perez-Liebana, Diego and Lucas, Simon M}, + booktitle={CIG}, + year={2016} +} + +@inproceedings{perez2016general, + title={General video game ai: Competition, challenges and opportunities}, + author={Perez-Liebana, Diego and Samothrakis, Spyridon and Togelius, Julian and Schaul, Tom and Lucas, Simon M}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2016} +} + +@phdthesis{marivate2015improved, + title={Improved empirical methods in reinforcement-learning evaluation}, + author={Marivate, Vukosi N}, + institution={Brown University}, + year={2015}, + school={Rutgers University-Graduate School-New Brunswick} +} + + +@inproceedings{whiteson2011protecting, + title={Protecting against evaluation overfitting in empirical reinforcement learning}, + author={Whiteson, Shimon and Tanner, Brian and Taylor, Matthew E and Stone, Peter}, + booktitle={ADPRL}, + year={2011} +} + +@article{vinyals2017starcraft, + title={Starcraft ii: A new challenge for reinforcement learning}, + author={Vinyals, Oriol and Ewalds, Timo and Bartunov, Sergey and Georgiev, Petko and Vezhnevets, Alexander Sasha and Yeo, Michelle and Makhzani, Alireza and K{\"u}ttler, Heinrich and Agapiou, John and Schrittwieser, Julian and others}, + journal={arXiv:1708.04782}, + year={2017} +} + +@book{sumpter2010collective, + title={Collective animal behavior}, + author={Sumpter, David JT}, + year={2010}, + publisher={Princeton University Press} +} + +@article{lazaridou2016multi, + title={Multi-agent cooperation and the emergence of (natural) language}, + author={Lazaridou, Angeliki and Peysakhovich, Alexander and Baroni, Marco}, + journal={arXiv:1612.07182}, + year={2016} +} + +@inproceedings{liu2019emergent, + title={Emergent Coordination Through Competition}, + author={Liu, Siqi and Lever, Guy and Merel, Josh and Tunyasuvunakool, Saran and Heess, Nicolas and Graepel, Thore}, + booktitle={ICLR}, + year={2019} +} + +@inproceedings{bansal2018emergent, + title={Emergent complexity via multi-agent competition}, + author={Bansal, Trapit and Pachocki, Jakub and Sidor, Szymon and Sutskever, Ilya and Mordatch, Igor}, + booktitle={International Conference on Learning Representations}, + year={2018} +} + +@article{leibo2019autocurricula, + title={Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research}, + author={Leibo, Joel Z and Hughes, Edward and Lanctot, Marc and Graepel, Thore}, + journal={arXiv:1903.00742}, + year={2019} +} + +@inproceedings{pathak2017curiosity, + title={Curiosity-driven exploration by self-supervised prediction}, + author={Pathak, Deepak and Agrawal, Pulkit and Efros, Alexei A and Darrell, Trevor}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2017} +} + +@misc{supplementary_material, + author = {Authors Anonymous}, + title = {Supplementary Material}, + howpublished = {\url{https://sites.google.com/view/arena-unity/home/supplementary}} , + year={2019} +} + +@inproceedings{goel2018unsupervised, + title={Unsupervised video object segmentation for deep reinforcement learning}, + author={Goel, Vikash and Weng, Jameson and Poupart, Pascal}, + booktitle={Advances in Neural Information Processing Systems}, + year={2018} +} + +@inproceedings{florensa2017stochastic, + title={Stochastic neural networks for hierarchical reinforcement learning}, + author={Florensa, Carlos and Duan, Yan and Abbeel, Pieter}, + booktitle={ICLR}, + year={2017} +} + +@article{friston2010free, + title={The free-energy principle: a unified brain theory?}, + author={Friston, Karl}, + journal={Nature Reviews Neuroscience}, + year={2010} +} + +@article{mnih2015human, + title={Human-level control through deep reinforcement learning}, + author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Rusu, Andrei A and Veness, Joel and Bellemare, Marc G and Graves, Alex and Riedmiller, Martin and Fidjeland, Andreas K and Ostrovski, Georg and others}, + journal={Nature}, + year={2015} +} + +@misc{OpenAI_false, + author = {OpenAI}, + title = {Faulty Reward Functions in the Wild}, + year = {2016} +} + +@inproceedings{burda2018exploration, + title={Exploration by random network distillation}, + author={Burda, Yuri and Edwards, Harrison and Storkey, Amos and Klimov, Oleg}, + booktitle={ICLR}, + year={2019} +} +@inproceedings{zambaldi2018deep, + title={Deep reinforcement learning with relational inductive biases}, + author={Zambaldi, Vinicius and Raposo, David and Santoro, Adam and Bapst, Victor and Li, Yujia and Babuschkin, Igor and Tuyls, Karl and Reichert, David and Lillicrap, Timothy and Lockhart, Edward and others}, + booktitle={ICLR}, + year={2019} +} + +@article{churchill2016starcraft, + title={StarCraft bots and competitions}, + author={Churchill, David and Preuss, Mike and Richoux, Florian and Synnaeve, Gabriel and Uriarte, Alberto and Onta{\~n}n{\'o}n, Santiago and {\v{C}}ertick{\`y}, Michal}, + journal={Encyclopedia of Computer Graphics and Games}, + year={2016}, +} + +@article{wymann2000torcs, + title={Torcs, the open racing car simulator}, + author={Wymann, Bernhard and Espi{\'e}, Eric and Guionneau, Christophe and Dimitrakakis, Christos and Coulom, R{\'e}mi and Sumner, Andrew}, + journal={Software available at http://torcs. sourceforge. net}, + year={2000} +} + +@inproceedings{wu2017evolutionary, + title={Evolutionary machine learning for RTS game StarCraft}, + author={Wu, Lianlong and Markham, Andrew}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2017} +} + +@inproceedings{Dosovitskiy17, + title = { {CARLA}: {An} Open Urban Driving Simulator}, + author = {Alexey Dosovitskiy and German Ros and Felipe Codevilla and Antonio Lopez and Vladlen Koltun}, + booktitle = {Proceedings of the 1st Annual Conference on Robot Learning}, + year = {2017} +} + +@inproceedings{handa2016scenenet, + title={Scenenet: An annotated model generator for indoor scene understanding}, + author={Handa, Ankur and P{\u{a}}tr{\u{a}}ucean, Viorica and Stent, Simon and Cipolla, Roberto}, + booktitle={International Conference on Robotics and Automation (ICRA)}, + year={2016} +} + +@inproceedings{ugur2015bottom, + title={Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning}, + author={Ugur, Emre and Piater, Justus}, + booktitle={International Conference on Robotics and Automation (ICRA)}, + year={2015} +} + +@inproceedings{Ha2018WorldM, + title={World Models}, + author={David Ha and J{\"u}rgen Schmidhuber}, + booktitle={Advances in Neural Information Processing Systems}, + year={2018} +} + +@inproceedings{zhao2017pyramid, + title={Pyramid scene parsing network}, + author={Zhao, Hengshuang and Shi, Jianping and Qi, Xiaojuan and Wang, Xiaogang and Jia, Jiaya}, + booktitle={CVPR}, + year={2017} +} + +@inproceedings{jaderberg2016reinforcement, + title={Reinforcement learning with unsupervised auxiliary tasks}, + author={Jaderberg, Max and Mnih, Volodymyr and Czarnecki, Wojciech Marian and Schaul, Tom and Leibo, Joel Z and Silver, David and Kavukcuoglu, Koray}, + booktitle={International Conference on Learning Representations}, + year={2017} +} + +@inproceedings{jin2018q, + title={Is q-learning provably efficient?}, + author={Jin, Chi and Allen-Zhu, Zeyuan and Bubeck, Sebastien and Jordan, Michael I}, + booktitle={Advances in Neural Information Processing Systems}, + pages={4863--4873}, + year={2018} +} + +@inproceedings{SWLXX-AAAI-2019, + title = "Diversity-Driven Extensible Hierarchical Reinforcement Learning", + author = "Yuhang Song and Jianyi Wang and Thomas Lukasiewicz and Zhenghua Xu and Mai Xu", + booktitle = "Proceedings of the AAAI Conference on Artificial Intelligence", + year = "2019", +} + +@inproceedings{ostrovski2017count, + title={Count-based exploration with neural density models}, + author={Ostrovski, Georg and Bellemare, Marc G and Oord, Aaron van den and Munos, R{\'e}mi}, + booktitle = "Proceedings of the International Conference on Machine Learning (ICML)", + year={2017} +} + +@inproceedings{tang2017exploration, + title={\# Exploration: A study of count-based exploration for deep reinforcement learning}, + author={Tang, Haoran and Houthooft, Rein and Foote, Davis and Stooke, Adam and Chen, Xi and Duan, Yan and Schulman, John and DeTurck, Filip and Abbeel, Pieter}, + booktitle={Advances in Neural Information Processing Systems}, + year={2017} +} + +@inproceedings{bellemare2016unifying, + title={Unifying count-based exploration and intrinsic motivation}, + author={Bellemare, Marc and Srinivasan, Sriram and Ostrovski, Georg and Schaul, Tom and Saxton, David and Munos, Remi}, + booktitle={Advances in Neural Information Processing Systems}, + year={2016} +} + +@article{watson1966development, + title={The development and generalization of" contingency awareness" in early infancy: Some hypotheses}, + author={Watson, John S}, + journal={Merrill-Palmer Quarterly of Behavior and Development}, + year={1966}, +} + +@article{amodei2016concrete, + title={Concrete problems in AI safety}, + author={Amodei, Dario and Olah, Chris and Steinhardt, Jacob and Christiano, Paul and Schulman, John and Man{\'e}, Dan}, + journal={arXiv:1606.06565}, + year={2016} +} + +@inproceedings{choi2018contingency, + title={Contingency-Aware Exploration in Reinforcement Learning}, + author={Choi, Jongwook and Guo, Yijie and Moczulski, Marcin and Oh, Junhyuk and Wu, Neal and Norouzi, Mohammad and Lee, Honglak}, + booktitle={ICLR}, + year={2019} +} + +@inproceedings{bellemare2012investigating, + title={Investigating Contingency Awareness Using Atari 2600 Games.}, + author={Bellemare, Marc G and Veness, Joel and Bowling, Michael}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2012} +} + +@inproceedings{martin2017count, + title={Count-based exploration in feature space for reinforcement learning}, + author={Martin, Jarryd and Sasikumar, Suraj Narayanan and Everitt, Tom and Hutter, Marcus}, + booktitle={International Joint Conference on Artificial Intelligence}, + year={2017} +} + +@article{moser2015place, + title={Place cells, grid cells, and memory}, + author={Moser, May-Britt and Rowland, David C and Moser, Edvard I}, + journal={Cold Spring Harbor perspectives in Biology}, + year={2015}, +} + +@article{banino2018vector, + title={Vector-based navigation using grid-like representations in artificial agents}, + author={Banino, Andrea and Barry, Caswell and Uria, Benigno and Blundell, Charles and Lillicrap, Timothy and Mirowski, Piotr and Pritzel, Alexander and Chadwick, Martin J and Degris, Thomas and Modayil, Joseph and others}, + journal={Nature}, + year={2018}, +} + +@article{baeyens1990contingency, + title={Contingency awareness in evaluative conditioning: A case for unaware affective-evaluative learning}, + author={Baeyens, Frank and Eelen, Paul and Bergh, Omer van den}, + journal={Cognition and emotion}, + year={1990} +} + +@article{oudeyer2009intrinsic, + title={What is intrinsic motivation? A typology of computational approaches}, + author={Oudeyer, Pierre-Yves and Kaplan, Frederic}, + journal={Frontiers in Neurorobotics}, + year={2009} +} + +@article{achiam2017surprise, + title={Surprise-based intrinsic motivation for deep reinforcement learning}, + author={Achiam, Joshua and Sastry, Shankar}, + journal={arXiv:1703.01732}, + year={2017} +} + +@inproceedings{sestini2019deepcrawl, + title={DeepCrawl: Deep Reinforcement Learning for Turn-based Strategy Games}, + author={Sestini, Alessandro and Kuhnle, Alexander and Bagdanov, Andrew D}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML) Workshop on Reinforcement Learning for Real Life}, + year={2019} +} + +@inproceedings{booth2019marathon, + title={Marathon Environments: Multi-Agent Continuous Control Benchmarks in a Modern Video Game Engine}, + author={Booth, Joe and Booth, Jackson}, + booktitle={AAAI Workshop on Games and Simulations for Artificial Intelligence}, + year={2019} +} + + +@inproceedings{burda2018large, + title={Large-scale study of curiosity-driven learning}, + author={Burda, Yuri and Edwards, Harri and Pathak, Deepak and Storkey, Amos and Darrell, Trevor and Efros, Alexei A}, + booktitle={Advances in Neural Information Processing Systems}, + year={2018} +} + +@inproceedings{bahdanau2014neural, + title={Neural machine translation by jointly learning to align and translate}, + author={Bahdanau, Dzmitry and Cho, Kyunghyun and Bengio, Yoshua}, + booktitle={ICLR}, + year={2015} +} + + +@inproceedings{xu2015show, + title={Show, attend and tell: Neural image caption generation with visual attention}, + author={Xu, Kelvin and Ba, Jimmy and Kiros, Ryan and Cho, Kyunghyun and Courville, Aaron and Salakhudinov, Ruslan and Zemel, Rich and Bengio, Yoshua}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + year={2015} +} + +@inproceedings{whittington2018generalisation, + title={Generalisation of structural knowledge in the Hippocampal-Entorhinal system}, + author={Whittington, James CR and Muller, Timothy H and Barry, Caswell and Behrens, Timothy EJ}, + booktitle={Advances in Neural Information Processing Systems}, + year={2018} +} + +@inproceedings{santoro2017simple, + title={A simple neural network module for relational reasoning}, + author={Santoro, Adam and Raposo, David and Barrett, David G and Malinowski, Mateusz and Pascanu, Razvan and Battaglia, Peter and Lillicrap, Tim}, + booktitle={Advances in Neural Information Processing Systems}, + year={2017} +} + +@inproceedings{mohamed2015variational, + title={Variational information maximisation for intrinsically motivated reinforcement learning}, + author={Mohamed, Shakir and Rezende, Danilo Jimenez}, + booktitle={Advances in Neural Information Processing Systems}, + year={2015} +} + +@article{klyubin2008keep, + title={Keep your options open: an information-based driving principle for sensorimotor systems}, + author={Klyubin, Alexander S and Polani, Daniel and Nehaniv, Chrystopher L}, + journal={PloS one}, + year={2008} +} + +@inproceedings{klyubin2005all, + title={All else being equal be empowered}, + author={Klyubin, Alexander S and Polani, Daniel and Nehaniv, Chrystopher L}, + booktitle={European Conferene on Artificial Life}, + year={2005} +} + +@inproceedings{watters2017visual, + title={Visual interaction networks: Learning a physics simulator from video}, + author={Watters, Nicholas and Zoran, Daniel and Weber, Theophane and Battaglia, Peter and Pascanu, Razvan and Tacchetti, Andrea}, + booktitle={Advances in Neural Information Processing Systems}, + year={2017} +} + +@article{montufar2016information, + title={Information theoretically aided reinforcement learning for embodied agents}, + author={Mont{\'u}far, Guido and Ghazi-Zahedi, Keyan and Ay, Nihat}, + journal={arXiv:1605.09735}, + year={2016} +} + +@inproceedings{talvitie2017self, + title={Self-Correcting Models for Model-Based Reinforcement Learning.}, + author={Talvitie, Erik}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + year={2017} +} + +@article{stadie2015incentivizing, + title={Incentivizing exploration in reinforcement learning with deep predictive models}, + author={Stadie, Bradly C and Levine, Sergey and Abbeel, Pieter}, + journal={arXiv:1507.00814}, + year={2015} +} + + + +% algorithms table +@TECHREPORT{Tsitsiklis97ananalysis, + author = {John N. Tsitsiklis and Benjamin Van Roy}, + title = {An analysis of temporal-difference learning with function approximation}, + institution = {IEEE Transactions on Automatic Control}, + year = {1997} +} + +@article{watkins1992q, + title={Q-learning}, + author={Watkins, Christopher JCH and Dayan, Peter}, + journal={Machine learning}, + volume={8}, + number={3-4}, + pages={279--292}, + year={1992}, + publisher={Springer} +} + +@book{rummery1994line, + title={On-line Q-learning using connectionist systems}, + author={Rummery, Gavin A and Niranjan, Mahesan}, + volume={37}, + year={1994}, + publisher={University of Cambridge, Department of Engineering Cambridge, England} +} +@inproceedings{lanctot2017unified, + title={A unified game-theoretic approach to multiagent reinforcement learning}, + author={Lanctot, Marc and Zambaldi, Vinicius and Gruslys, Audrunas and Lazaridou, Angeliki and Tuyls, Karl and P{\'e}rolat, Julien and Silver, David and Graepel, Thore}, + booktitle={Advances in Neural Information Processing Systems}, + pages={4190--4203}, + year={2017} +} +@article{berner2019dota, + title={Dota 2 with Large Scale Deep Reinforcement Learning}, + author={Berner, Christopher and Brockman, Greg and Chan, Brooke and Cheung, Vicki and D{\k{e}}biak, Przemys{\l}aw and Dennison, Christy and Farhi, David and Fischer, Quirin and Hashme, Shariq and Hesse, Chris and others}, + journal={arXiv preprint arXiv:1912.06680}, + year={2019} +} + + +@incollection{nowe2012game, + title={Game theory and multi-agent reinforcement learning}, + author={Now{\'e}, Ann and Vrancx, Peter and De Hauwere, Yann-Micha{\"e}l}, + booktitle={Reinforcement Learning}, + pages={441--470}, + year={2012}, + publisher={Springer} +} + + + +@inproceedings{tan1993multi, + title={Multi-agent reinforcement learning: Independent vs. cooperative agents}, + author={Tan, Ming}, + booktitle={Proceedings of the tenth International Conference on Machine Learning (ICML)}, + pages={330--337}, + year={1993} +} + +@inproceedings{mnih2016asynchronous, + title={Asynchronous methods for deep reinforcement learning}, + author={Mnih, Volodymyr and Badia, Adria Puigdomenech and Mirza, Mehdi and Graves, Alex and Lillicrap, Timothy and Harley, Tim and Silver, David and Kavukcuoglu, Koray}, + booktitle={International Conference on Machine Learning (ICML)}, + pages={1928--1937}, + year={2016} +} +@article{vinyals2019grandmaster, + title={Grandmaster level in StarCraft II using multi-agent reinforcement learning}, + author={Vinyals, Oriol and Babuschkin, Igor and Czarnecki, Wojciech M and Mathieu, Micha{\"e}l and Dudzik, Andrew and Chung, Junyoung and Choi, David H and Powell, Richard and Ewalds, Timo and Georgiev, Petko and others}, + journal={Nature}, + volume={575}, + number={7782}, + pages={350--354}, + year={2019}, + publisher={Nature Publishing Group} +} + +@article{szita2006learning, + title={Learning Tetris using the noisy cross-entropy method}, + author={Szita, Istv{\'a}n and L{\"o}rincz, Andr{\'a}s}, + journal={Neural computation}, + volume={18}, + number={12}, + pages={2936--2941}, + year={2006}, + publisher={MIT Press} +} + + + +@article{jaderberg2017population, + title={Population based training of neural networks}, + author={Jaderberg, Max and Dalibard, Valentin and Osindero, Simon and Czarnecki, Wojciech M and Donahue, Jeff and Razavi, Ali and Vinyals, Oriol and Green, Tim and Dunning, Iain and Simonyan, Karen and others}, + journal={arXiv preprint arXiv:1711.09846}, + year={2017} +} + + +@article{silver2018general, + title={A general reinforcement learning algorithm that masters chess, shogi, and {G}o through self-play}, + author={Silver, David and Hubert, Thomas and Schrittwieser, Julian and Antonoglou, Ioannis and Lai, Matthew and Guez, Arthur and Lanctot, Marc and Sifre, Laurent and Kumaran, Dharshan and Graepel, Thore and others}, + journal={Science}, + volume={362}, + number={6419}, + pages={1140--1144}, + year={2018}, + publisher={American Association for the Advancement of Science} +} + +@article{heinrich2016deep, + title={Deep reinforcement learning from self-play in imperfect-information games}, + author={Heinrich, Johannes and Silver, David}, + journal={arXiv preprint arXiv:1603.01121}, + year={2016} +} + +@article{shoham2003multi, + title={Multi-agent reinforcement learning: a critical survey}, + author={Shoham, Yoav and Powers, Rob and Grenager, Trond}, + journal={Web manuscript}, + year={2003} +} + + + +@article{lillicrap2015continuous, + title={Continuous control with deep reinforcement learning}, + author={Lillicrap, Timothy P and Hunt, Jonathan J and Pritzel, Alexander and Heess, Nicolas and Erez, Tom and Tassa, Yuval and Silver, David and Wierstra, Daan}, + journal={arXiv preprint arXiv:1509.02971}, + year={2015} +} + +@article{fujimoto2018addressing, + title={Addressing function approximation error in actor-critic methods}, + author={Fujimoto, Scott and van Hoof, Herke and Meger, David}, + journal={arXiv preprint arXiv:1802.09477}, + year={2018} +} + +@article{haarnoja2018soft, + title={Soft actor-critic algorithms and applications}, + author={Haarnoja, Tuomas and Zhou, Aurick and Hartikainen, Kristian and Tucker, George and Ha, Sehoon and Tan, Jie and Kumar, Vikash and Zhu, Henry and Gupta, Abhishek and Abbeel, Pieter and others}, + journal={arXiv preprint arXiv:1812.05905}, + year={2018} +} +@inproceedings{martens2015optimizing, + title={Optimizing neural networks with kronecker-factored approximate curvature}, + author={Martens, James and Grosse, Roger}, + booktitle={International Conference on Machine Learning (ICML)}, + pages={2408--2417}, + year={2015} +} +@inproceedings{grosse2016kronecker, + title={A kronecker-factored approximate fisher matrix for convolution layers}, + author={Grosse, Roger and Martens, James}, + booktitle={International Conference on Machine Learning (ICML)}, + pages={573--582}, + year={2016} +} +@inproceedings{schulman2015trust, + title={Trust region policy optimization}, + author={Schulman, John and Levine, Sergey and Abbeel, Pieter and Jordan, Michael and Moritz, Philipp}, + booktitle={International Conference on Machine Learning (ICML)}, + pages={1889--1897}, + year={2015} +} + +@article{schulman2017proximal, + title={Proximal policy optimization algorithms}, + author={Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg}, + journal={arXiv preprint arXiv:1707.06347}, + year={2017} +} + +@inproceedings{wu2017scalable, + title={Scalable trust-region method for deep reinforcement learning using kronecker-factored approximation}, + author={Wu, Yuhuai and Mansimov, Elman and Grosse, Roger B and Liao, Shun and Ba, Jimmy}, + booktitle={Advances in Neural Information Processing Systems}, + pages={5279--5288}, + year={2017} +} + + +@article{amari1998natural, + title={Natural gradient works efficiently in learning}, + author={Amari, Shun-Ichi}, + journal={Neural computation}, + volume={10}, + number={2}, + pages={251--276}, + year={1998}, + publisher={MIT Press} +} + + +@inproceedings{williams1988use, + title={On the use of backpropagation in associative reinforcement learning}, + author={Williams, Ronald J}, + booktitle={Proceedings of the IEEE International Conference on Neural Networks}, + volume={1}, + pages={263--270}, + year={1988}, + organization={San Diego, CA.} +} + +@inproceedings{kakade2002approximately, + title={Approximately optimal approximate reinforcement learning}, + author={Kakade, Sham and Langford, John}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + volume={2}, + pages={267--274}, + year={2002} +} + +@article{coumans2013bullet, + title={Bullet physics library}, + author={Coumans, Erwin and others}, + journal={Open source: bulletphysics. org}, + volume={15}, + number={49}, + pages={5}, + year={2013} +} + +@article{michel2004cyberbotics, + title={Cyberbotics Ltd. Webots™: professional mobile robot simulation}, + author={Michel, Olivier}, + journal={International Journal of Advanced Robotic Systems}, + volume={1}, + number={1}, + pages={5}, + year={2004}, + publisher={SAGE Publications Sage UK: London, England} +} + + +@inproceedings{todorov2012mujoco, + title={Mujoco: A physics engine for model-based control}, + author={Todorov, Emanuel and Erez, Tom and Tassa, Yuval}, + booktitle={2012 IEEE/RSJ International Conference on Intelligent Robots and Systems}, + pages={5026--5033}, + year={2012}, + organization={IEEE} +} + + +@article{coumans2016pybullet, + title={Pybullet, a python module for physics simulation for games, robotics and machine learning}, + author={Coumans, Erwin and Bai, Yunfei}, + journal={GitHub repository}, + year={2016} +} + + + +@inproceedings{rohmer2013v, + title={V-REP: A versatile and scalable robot simulation framework}, + author={Rohmer, Eric and Singh, Surya PN and Freese, Marc}, + booktitle={2013 IEEE/RSJ International Conference on Intelligent Robots and Systems}, + pages={1321--1326}, + year={2013}, + organization={IEEE} +} + +@article{james2019pyrep, + title={PyRep: Bringing V-REP to Deep Robot Learning}, + author={James, Stephen and Freese, Marc and Davison, Andrew J}, + journal={arXiv preprint arXiv:1906.11176}, + year={2019} +} + + + + +@article{rubinstein2004cross, + title={The cross-entropy method: A unified approach to Monte Carlo simulation, randomized optimization and machine learning}, + author={Rubinstein, Reuven Y and Kroese, Dirk P}, + journal={Information Science \& Statistics, Springer Verlag, NY}, + year={2004} +} + +@article{leshno1993multilayer, + title={Multilayer feedforward networks with a nonpolynomial activation function can approximate any function}, + author={Leshno, Moshe and Lin, Vladimir Ya and Pinkus, Allan and Schocken, Shimon}, + journal={Neural networks}, + volume={6}, + number={6}, + pages={861--867}, + year={1993}, + publisher={Elsevier} +} + +@article{wang2015dueling, + title={Dueling network architectures for deep reinforcement learning}, + author={Wang, Ziyu and Schaul, Tom and Hessel, Matteo and Van Hasselt, Hado and Lanctot, Marc and De Freitas, Nando}, + journal={arXiv preprint arXiv:1511.06581}, + year={2015} +} + +@inproceedings{lattimore2013sample, + title={The sample-complexity of general reinforcement learning}, + author={Lattimore, Tor and Hutter, Marcus and Sunehag, Peter and others}, + booktitle={Proceedings of the 30th International Conference on Machine Learning}, + year={2013}, + organization={Journal of Machine Learning Research} +} + +@inproceedings{koenig1993complexity, + title={Complexity analysis of real-time reinforcement learning}, + author={Koenig, Sven and Simmons, Reid G}, + booktitle={Proceedings of the AAAI Conference on Artificial Intelligence}, + pages={99--107}, + year={1993} +} + + +% challenge: sim to real +@inproceedings{ramstedt2019real, + title={Real-Time Reinforcement Learning}, + author={Ramstedt, Simon and Pal, Chris}, + booktitle={Advances in Neural Information Processing Systems}, + pages={3067--3076}, + year={2019} +} + +@article{jeong2019modelling, + title={Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer}, + author={Jeong, Rae and Kay, Jackie and Romano, Francesco and Lampe, Thomas and Rothorl, Tom and Abdolmaleki, Abbas and Erez, Tom and Tassa, Yuval and Nori, Francesco}, + journal={arXiv preprint arXiv:1910.09471}, + year={2019} +} +@article{haarnoja2018soft, + title={Soft actor-critic algorithms and applications}, + author={Haarnoja, Tuomas and Zhou, Aurick and Hartikainen, Kristian and Tucker, George and Ha, Sehoon and Tan, Jie and Kumar, Vikash and Zhu, Henry and Gupta, Abhishek and Abbeel, Pieter and others}, + journal={arXiv preprint arXiv:1812.05905}, + year={2018} +} + +@article{jeong2019self, + title={Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation}, + author={Jeong, Rae and Aytar, Yusuf and Khosid, David and Zhou, Yuxiang and Kay, Jackie and Lampe, Thomas and Bousmalis, Konstantinos and Nori, Francesco}, + journal={arXiv preprint arXiv:1910.09470}, + year={2019} +} + +@article{kirkpatrick2017overcoming, + title={Overcoming catastrophic forgetting in neural networks}, + author={Kirkpatrick, James and Pascanu, Razvan and Rabinowitz, Neil and Veness, Joel and Desjardins, Guillaume and Rusu, Andrei A and Milan, Kieran and Quan, John and Ramalho, Tiago and Grabska-Barwinska, Agnieszka and others}, + journal={Proceedings of the national academy of sciences}, + volume={114}, + number={13}, + pages={3521--3526}, + year={2017}, + publisher={National Acad Sciences} +} + +@article{torabi2019recent, + title={Recent Advances in Imitation Learning from Observation}, + author={Torabi, Faraz and Warnell, Garrett and Stone, Peter}, + journal={arXiv preprint arXiv:1905.13566}, + year={2019} +} + +@inproceedings{finn2017model, + title={Model-agnostic meta-learning for fast adaptation of deep networks}, + author={Finn, Chelsea and Abbeel, Pieter and Levine, Sergey}, + booktitle={Proceedings of the 34th International Conference on Machine Learning-Volume 70}, + pages={1126--1135}, + year={2017}, + organization={JMLR. org} +} + +@book{bengio1990learning, + title={Learning a synaptic learning rule}, + author={Bengio, Yoshua and Bengio, Samy and Cloutier, Jocelyn}, + year={1990}, + publisher={Universit{\'e} de Montr{\'e}al, D{\'e}partement d'informatique et de recherche op{\'e}rationnelle} +} + +@article{bengio2013representation, + title={Representation learning: A review and new perspectives}, + author={Bengio, Yoshua and Courville, Aaron and Vincent, Pascal}, + journal={IEEE transactions on pattern analysis and machine intelligence}, + volume={35}, + number={8}, + pages={1798--1828}, + year={2013}, + publisher={IEEE} +} + +@article{vinyals2019grandmaster, + title={Grandmaster level in StarCraft II using multi-agent reinforcement learning}, + author={Vinyals, Oriol and Babuschkin, Igor and Czarnecki, Wojciech M and Mathieu, Micha{\"e}l and Dudzik, Andrew and Chung, Junyoung and Choi, David H and Powell, Richard and Ewalds, Timo and Georgiev, Petko and others}, + journal={Nature}, + pages={1--5}, + year={2019}, + publisher={Nature Publishing Group} +} + +@inproceedings{aytar2018playing, + title={Playing hard exploration games by watching youtube}, + author={Aytar, Yusuf and Pfaff, Tobias and Budden, David and Paine, Thomas and Wang, Ziyu and de Freitas, Nando}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2930--2941}, + year={2018} +} + +@inproceedings{houthooft2016vime, + title={Vime: Variational information maximizing exploration}, + author={Houthooft, Rein and Chen, Xi and Duan, Yan and Schulman, John and De Turck, Filip and Abbeel, Pieter}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1109--1117}, + year={2016} +} + +@article{heess2016learning, + title={Learning and transfer of modulated locomotor controllers}, + author={Heess, Nicolas and Wayne, Greg and Tassa, Yuval and Lillicrap, Timothy and Riedmiller, Martin and Silver, David}, + journal={arXiv preprint arXiv:1610.05182}, + year={2016} +} + +@inproceedings{andrychowicz2017hindsight, + title={Hindsight experience replay}, + author={Andrychowicz, Marcin and Wolski, Filip and Ray, Alex and Schneider, Jonas and Fong, Rachel and Welinder, Peter and McGrew, Bob and Tobin, Josh and Abbeel, OpenAI Pieter and Zaremba, Wojciech}, + booktitle={Advances in Neural Information Processing Systems}, + pages={5048--5058}, + year={2017} +} + +@article{mnih2013playing, + title={Playing atari with deep reinforcement learning}, + author={Mnih, Volodymyr and Kavukcuoglu, Koray and Silver, David and Graves, Alex and Antonoglou, Ioannis and Wierstra, Daan and Riedmiller, Martin}, + journal={arXiv preprint arXiv:1312.5602}, + year={2013} +} + +@article{fujimoto2018addressing, + title={Addressing function approximation error in actor-critic methods}, + author={Fujimoto, Scott and van Hoof, Herke and Meger, David}, + journal={arXiv preprint arXiv:1802.09477}, + year={2018} +} + +@inproceedings{deisenroth2011pilco, + title={PILCO: A model-based and data-efficient approach to policy search}, + author={Deisenroth, Marc and Rasmussen, Carl E}, + booktitle={Proceedings of the 28th International Conference on Machine Learning (ICML-11)}, + pages={465--472}, + year={2011} +} + +@article{hafner2019dream, + title={Dream to Control: Learning Behaviors by Latent Imagination}, + author={Hafner, Danijar and Lillicrap, Timothy and Ba, Jimmy and Norouzi, Mohammad}, + journal={arXiv preprint arXiv:1912.01603}, + year={2019} +} + + +@article{mahmood2018benchmarking, + title={Benchmarking reinforcement learning algorithms on real-world robots}, + author={Mahmood, A Rupam and Korenkevych, Dmytro and Vasan, Gautham and Ma, William and Bergstra, James}, + journal={arXiv preprint arXiv:1809.07731}, + year={2018} +} + +@inproceedings{levine2013guided, + title={Guided policy search}, + author={Levine, Sergey and Koltun, Vladlen}, + booktitle={International Conference on Machine Learning}, + pages={1--9}, + year={2013} +} + +@article{levine2018learning, + title={Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection}, + author={Levine, Sergey and Pastor, Peter and Krizhevsky, Alex and Ibarz, Julian and Quillen, Deirdre}, + journal={The International Journal of Robotics Research}, + volume={37}, + number={4-5}, + pages={421--436}, + year={2018}, + publisher={SAGE Publications Sage UK: London, England} +} + +@inproceedings{peng2018sim, + title={Sim-to-real transfer of robotic control with dynamics randomization}, + author={Peng, Xue Bin and Andrychowicz, Marcin and Zaremba, Wojciech and Abbeel, Pieter}, + booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={1--8}, + year={2018}, + organization={IEEE} +} + +@article{sadeghi2016cad2rl, + title={Cad2rl: Real single-image flight without a single real image}, + author={Sadeghi, Fereshteh and Levine, Sergey}, + journal={arXiv preprint arXiv:1611.04201}, + year={2016} +} + +@inproceedings{tobin2017domain, + title={Domain randomization for transferring deep neural networks from simulation to the real world}, + author={Tobin, Josh and Fong, Rachel and Ray, Alex and Schneider, Jonas and Zaremba, Wojciech and Abbeel, Pieter}, + booktitle={2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, + pages={23--30}, + year={2017}, + organization={IEEE} +} + +@inproceedings{james2019sim, + title={Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks}, + author={James, Stephen and Wohlhart, Paul and Kalakrishnan, Mrinal and Kalashnikov, Dmitry and Irpan, Alex and Ibarz, Julian and Levine, Sergey and Hadsell, Raia and Bousmalis, Konstantinos}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={12627--12637}, + year={2019} +} + +@article{rusu2016sim, + title={Sim-to-real robot learning from pixels with progressive nets}, + author={Rusu, Andrei A and Vecerik, Mel and Roth{\"o}rl, Thomas and Heess, Nicolas and Pascanu, Razvan and Hadsell, Raia}, + journal={arXiv preprint arXiv:1610.04286}, + year={2016} +} + +@article{rusu2016progressive, + title={Progressive neural networks}, + author={Rusu, Andrei A and Rabinowitz, Neil C and Desjardins, Guillaume and Soyer, Hubert and Kirkpatrick, James and Kavukcuoglu, Koray and Pascanu, Razvan and Hadsell, Raia}, + journal={arXiv preprint arXiv:1606.04671}, + year={2016} +} + +@article{james2019rlbench, + title={RLBench: The Robot Learning Benchmark \& Learning Environment}, + author={James, Stephen and Ma, Zicong and Arrojo, David Rovick and Davison, Andrew J}, + journal={arXiv preprint arXiv:1909.12271}, + year={2019} +} +@inproceedings{degris2012linear, + title={Linear off-policy actor-critic}, + author={Degris, Thomas and White, Martha and Sutton, Richard S}, + booktitle={In International Conference on Machine Learning}, + year={2012}, + organization={Citeseer} +} + +@misc{clevert2015fast, + title={Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)}, + author={Djork-Arné Clevert and Thomas Unterthiner and Sepp Hochreiter}, + year={2015}, + eprint={1511.07289}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@misc{agarap2018deep, + title={Deep Learning using Rectified Linear Units (ReLU)}, + author={Abien Fred Agarap}, + year={2018}, + eprint={1803.08375}, + archivePrefix={arXiv}, + primaryClass={cs.NE} +} + +@misc{ba2016layer, + title={Layer Normalization}, + author={Jimmy Lei Ba and Jamie Ryan Kiros and Geoffrey E. Hinton}, + year={2016}, + eprint={1607.06450}, + archivePrefix={arXiv}, + primaryClass={stat.ML} +} + + +% implementation trick +@article{heess2015memory, + title={Memory-based control with recurrent neural networks}, + author={Heess, Nicolas and Hunt, Jonathan J and Lillicrap, Timothy P and Silver, David}, + journal={arXiv preprint arXiv:1512.04455}, + year={2015} +} + +@article{espeholt2018impala, + title={Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures}, + author={Espeholt, Lasse and Soyer, Hubert and Munos, Remi and Simonyan, Karen and Mnih, Volodymir and Ward, Tom and Doron, Yotam and Firoiu, Vlad and Harley, Tim and Dunning, Iain and others}, + journal={arXiv preprint arXiv:1802.01561}, + year={2018} +} + +@inproceedings{fu2018variational, + title={Variational inverse control with events: A general framework for data-driven reward definition}, + author={Fu, Justin and Singh, Avi and Ghosh, Dibya and Yang, Larry and Levine, Sergey}, + booktitle={Advances in Neural Information Processing Systems}, + pages={8538--8547}, + year={2018} +} + +@article{saxe2013exact, + title={Exact solutions to the nonlinear dynamics of learning in deep linear neural networks}, + author={Saxe, Andrew M and McClelland, James L and Ganguli, Surya}, + journal={arXiv preprint arXiv:1312.6120}, + year={2013} +} + +@inproceedings{glorot2010understanding, + title={Understanding the difficulty of training deep feedforward neural networks}, + author={Glorot, Xavier and Bengio, Yoshua}, + booktitle={Proceedings of the thirteenth international conference on artificial intelligence and statistics}, + pages={249--256}, + year={2010} +} + +@article{levine2018reinforcement, + title={Reinforcement learning and control as probabilistic inference: Tutorial and review}, + author={Levine, Sergey}, + journal={arXiv preprint arXiv:1805.00909}, + year={2018} +} + +% control as inference +@article{levine2018reinforcement, + title={Reinforcement learning and control as probabilistic inference: Tutorial and review}, + author={Levine, Sergey}, + journal={arXiv preprint arXiv:1805.00909}, + year={2018} +} + +@inproceedings{fu2018variational, + title={Variational inverse control with events: A general framework for data-driven reward definition}, + author={Fu, Justin and Singh, Avi and Ghosh, Dibya and Yang, Larry and Levine, Sergey}, + booktitle={Advances in Neural Information Processing Systems}, + pages={8538--8547}, + year={2018} +} + + +@inproceedings{sutton2000policy, + title={Policy gradient methods for reinforcement learning with function approximation}, + author={Sutton, Richard S and McAllester, David A and Singh, Satinder P and Mansour, Yishay}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1057--1063}, + year={2000} +} + +@article{peters2008natural, + title={Natural actor-critic}, + author={Peters, Jan and Schaal, Stefan}, + journal={Neurocomputing}, + volume={71}, + number={7-9}, + pages={1180--1190}, + year={2008}, + publisher={Elsevier} +} + +@article{kalashnikov2018qt, + title={Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation}, + author={Kalashnikov, Dmitry and Irpan, Alex and Pastor, Peter and Ibarz, Julian and Herzog, Alexander and Jang, Eric and Quillen, Deirdre and Holly, Ethan and Kalakrishnan, Mrinal and Vanhoucke, Vincent and others}, + journal={arXiv preprint arXiv:1806.10293}, + year={2018} +} + +@inproceedings{silver2014deterministic, + title={Deterministic policy gradient algorithms}, + author={Silver, David and Lever, Guy and Heess, Nicolas and Degris, Thomas and Wierstra, Daan and Riedmiller, Martin}, + year={2014} +} + +@article{bodnar2019quantile, + title={Quantile {QT}-{O}pt for Risk-Aware Vision-Based Robotic Grasping}, + author={Bodnar, Cristian and Li, Adrian and Hausman, Karol and Pastor, Peter and Kalakrishnan, Mrinal}, + journal={arXiv preprint arXiv:1910.02787}, + year={2019} +} + + +@article{schmidhuber2015deep, + title={Deep learning in neural networks: An overview}, + author={Schmidhuber, J{\"u}rgen}, + journal={Neural networks}, + volume={61}, + pages={85--117}, + year={2015}, + publisher={Elsevier} +} + +@inproceedings{pyeatt2001decision, + title={Decision tree function approximation in reinforcement learning}, + author={Pyeatt, Larry D and Howe, Adele E and others}, + booktitle={Proceedings of the third international symposium on adaptive systems: evolutionary computation and probabilistic graphical models}, + volume={2}, + number={1/2}, + pages={70--77}, + year={2001}, + organization={Cuba} +} + +@article{achiam2019towards, + title={Towards Characterizing Divergence in Deep Q-Learning}, + author={Achiam, Joshua and Knight, Ethan and Abbeel, Pieter}, + journal={arXiv preprint arXiv:1903.08894}, + year={2019} +} + +@article{van2018deep, + title={Deep reinforcement learning and the deadly triad}, + author={Van Hasselt, Hado and Doron, Yotam and Strub, Florian and Hessel, Matteo and Sonnerat, Nicolas and Modayil, Joseph}, + journal={arXiv preprint arXiv:1812.02648}, + year={2018} +} + +@article{yu2017preparing, + title={Preparing for the unknown: Learning a universal policy with online system identification}, + author={Yu, Wenhao and Tan, Jie and Liu, C Karen and Turk, Greg}, + journal={arXiv preprint arXiv:1702.02453}, + year={2017} +} + + +@article{zhou2019environment, + title={Environment probing interaction policies}, + author={Zhou, Wenxuan and Pinto, Lerrel and Gupta, Abhinav}, + journal={arXiv preprint arXiv:1907.11740}, + year={2019} +} + + + +@inproceedings{duan2016benchmarking, + title={Benchmarking deep reinforcement learning for continuous control}, + author={Duan, Yan and Chen, Xi and Houthooft, Rein and Schulman, John and Abbeel, Pieter}, + booktitle={International Conference on Machine Learning}, + pages={1329--1338}, + year={2016} +} + +@inproceedings{heess2015learning, + title={Learning continuous control policies by stochastic value gradients}, + author={Heess, Nicolas and Wayne, Gregory and Silver, David and Lillicrap, Timothy and Erez, Tom and Tassa, Yuval}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2944--2952}, + year={2015} +} + +@article{silver2018residual, + title={Residual policy learning}, + author={Silver, Tom and Allen, Kelsey and Tenenbaum, Josh and Kaelbling, Leslie}, + journal={arXiv preprint arXiv:1812.06298}, + year={2018} +} + +@inproceedings{johannink2019residual, + title={Residual reinforcement learning for robot control}, + author={Johannink, Tobias and Bahl, Shikhar and Nair, Ashvin and Luo, Jianlan and Kumar, Avinash and Loskyll, Matthias and Ojea, Juan Aparicio and Solowjow, Eugen and Levine, Sergey}, + booktitle={2019 International Conference on Robotics and Automation (ICRA)}, + pages={6023--6029}, + year={2019}, + organization={IEEE} +} + + + +@article{nagabandi2018learning, + title={Learning to adapt in dynamic, real-world environments through meta-reinforcement learning}, + author={Nagabandi, Anusha and Clavera, Ignasi and Liu, Simin and Fearing, Ronald S and Abbeel, Pieter and Levine, Sergey and Finn, Chelsea}, + journal={arXiv preprint arXiv:1803.11347}, + year={2018} +} + + +@article{arndt2019meta, + title={Meta Reinforcement Learning for Sim-to-real Domain Adaptation}, + author={Arndt, Karol and Hazara, Murtaza and Ghadirzadeh, Ali and Kyrki, Ville}, + journal={arXiv preprint arXiv:1909.12906}, + year={2019} +} + + +@inproceedings{james2019sim, + title={Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks}, + author={James, Stephen and Wohlhart, Paul and Kalakrishnan, Mrinal and Kalashnikov, Dmitry and Irpan, Alex and Ibarz, Julian and Levine, Sergey and Hadsell, Raia and Bousmalis, Konstantinos}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={12627--12637}, + year={2019} +} + + + +% rl for robotics +@article{akkaya2019solving, + title={Solving Rubik's Cube with a Robot Hand}, + author={Akkaya, Ilge and Andrychowicz, Marcin and Chociej, Maciek and Litwin, Mateusz and McGrew, Bob and Petron, Arthur and Paino, Alex and Plappert, Matthias and Powell, Glenn and Ribas, Raphael and others}, + journal={arXiv preprint arXiv:1910.07113}, + year={2019} +} + +@article{andrychowicz2018learning, + title={Learning dexterous in-hand manipulation}, + author={Andrychowicz, Marcin and Baker, Bowen and Chociej, Maciek and Jozefowicz, Rafal and McGrew, Bob and Pachocki, Jakub and Petron, Arthur and Plappert, Matthias and Powell, Glenn and Ray, Alex and others}, + journal={arXiv preprint arXiv:1808.00177}, + year={2018} +} + +@article{korenkevych2019autoregressive, + title={Autoregressive Policies for Continuous Control Deep Reinforcement Learning}, + author={Korenkevych, Dmytro and Mahmood, A Rupam and Vasan, Gautham and Bergstra, James}, + journal={arXiv preprint arXiv:1903.11524}, + year={2019} +} + + +@article{li2017deep, + title={Deep reinforcement learning: An overview}, + author={Li, Yuxi}, + journal={arXiv preprint arXiv:1701.07274}, + year={2017} +} + +@article{haarnoja2018softactorcritic, + title={Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor}, + author={Haarnoja, Tuomas and Zhou, Aurick and Abbeel, Pieter and Levine, Sergey}, + journal={arXiv preprint arXiv:1801.01290}, + year={2018} +} + +@article{browne2012survey, + title={A survey of monte carlo tree search methods}, + author={Browne, Cameron B and Powley, Edward and Whitehouse, Daniel and Lucas, Simon M and Cowling, Peter I and Rohlfshagen, Philipp and Tavener, Stephen and Perez, Diego and Samothrakis, Spyridon and Colton, Simon}, + journal={IEEE Transactions on Computational Intelligence and AI in games}, + volume={4}, + number={1}, + pages={1--43}, + year={2012}, + publisher={IEEE} +} + +@inproceedings{ha2018recurrent, + title={Recurrent world models facilitate policy evolution}, + author={Ha, David and Schmidhuber, J{\"u}rgen}, + booktitle={Advances in Neural Information Processing Systems}, + pages={2450--2462}, + year={2018} +} + +@inproceedings{racaniere2017imagination, + title={Imagination-augmented agents for deep reinforcement learning}, + author={Racani{\`e}re, S{\'e}bastien and Weber, Th{\'e}ophane and Reichert, David and Buesing, Lars and Guez, Arthur and Rezende, Danilo Jimenez and Badia, Adria Puigdom{\`e}nech and Vinyals, Oriol and Heess, Nicolas and Li, Yujia and others}, + booktitle={Advances in Neural Information Processing Systems}, + pages={5690--5701}, + year={2017} +} + +@inproceedings{kocsis2006bandit, + title={Bandit based monte-carlo planning}, + author={Kocsis, Levente and Szepesv{\'a}ri, Csaba}, + booktitle={European conference on machine learning}, + pages={282--293}, + year={2006}, + organization={Springer} +} + +@inproceedings{nagabandi2018neural, + title={Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning}, + author={Nagabandi, Anusha and Kahn, Gregory and Fearing, Ronald S and Levine, Sergey}, + booktitle={2018 IEEE International Conference on Robotics and Automation (ICRA)}, + pages={7559--7566}, + year={2018}, + organization={IEEE} +} + +@article{feinberg2018model, + title={Model-based value estimation for efficient model-free reinforcement learning}, + author={Feinberg, Vladimir and Wan, Alvin and Stoica, Ion and Jordan, Michael I and Gonzalez, Joseph E and Levine, Sergey}, + journal={arXiv preprint arXiv:1803.00101}, + year={2018} +} + +@inproceedings{Cou2011ContinuousEstimates, + title={Continuous rapid action value estimates}, + author={Couetoux, Adrien and Milone, Mario and Brendel, Matyas and Doghmen, Hassan and Sebag, Michele and Teytaud, Olivier}, + booktitle={Asian Conference on Machine Learning}, + pages={19--31}, + year={2011} +} + +@article{watkins1992q, + title={Q-learning}, + author={Watkins, Christopher JCH and Dayan, Peter}, + journal={Machine learning}, + volume={8}, + number={3-4}, + pages={279--292}, + year={1992}, + publisher={Springer} +} +% Chapter learning and planning +@article{sutton1991dyna, + title={Dyna, an integrated architecture for learning, planning, and reacting}, + author={Sutton, Richard S}, + journal={ACM Sigart Bulletin}, + volume={2}, + number={4}, + pages={160--163}, + year={1991}, + publisher={ACM} +} + + +@article{silver2012temporal, + title={Temporal-difference search in computer Go}, + author={Silver, David and Sutton, Richard S and M{\"u}ller, Martin}, + journal={Machine learning}, + volume={87}, + number={2}, + pages={183--219}, + year={2012}, + publisher={Springer} +} + +@inproceedings{silver2008sample, + title={Sample-based learning and search with permanent and transient memories}, + author={Silver, David and Sutton, Richard S and M{\"u}ller, Martin}, + booktitle={Proceedings of the 25th international conference on Machine learning}, + pages={968--975}, + year={2008}, + organization={ACM} +} + +@misc{kaiser2019modelbased, + title={Model-Based Reinforcement Learning for Atari}, + author={Lukasz Kaiser and Mohammad Babaeizadeh and Piotr Milos and Blazej Osinski and Roy H Campbell and Konrad Czechowski and Dumitru Erhan and Chelsea Finn and Piotr Kozakowski and Sergey Levine and Afroz Mohiuddin and Ryan Sepassi and George Tucker and Henryk Michalewski}, + year={2019}, + eprint={1903.00374}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +% Chapter MARL +@inproceedings{Littman1994MARLfootball, + author={Michael L. Littman}, + title={Markov Games as a Framework for Multi-Agent Reinforcement Learning}, + year={1994}, + cdate={757382400000}, + pages={157-163}, + url={https://doi.org/10.1016/b978-1-55860-335-6.50027-1}, + booktitle={Proceedings of the International Conference on Machine Learning (ICML)}, + crossref={conf/icml/1994} +} + +@inproceedings{Hu1998MultiagentRL, + title={Multiagent Reinforcement Learning: Theoretical Framework and an Algorithm}, + author={Junling Hu and Michael P. Wellman}, + booktitle={International Conference on Robotics and Automation (ICRA)}, + year={1998} +} + +@article{fudenberg1991game, + title={Game theory, 1991}, + author={Fudenberg, Drew and Tirole, Jean}, + journal={Cambridge, Massachusetts}, + volume={393}, + number={12}, + pages={80}, + year={1991} +} + +@article{nash1950equilibrium, + title={Equilibrium points in n-person games}, + author={Nash, John F and others}, + journal={Proceedings of the national academy of sciences}, + volume={36}, + number={1}, + pages={48--49}, + year={1950}, + publisher={USA} +} + +@article{rapoport1966game, + title={The game of chicken}, + author={Rapoport, Anatol and Chammah, Albert M}, + journal={American Behavioral Scientist}, + volume={10}, + number={3}, + pages={10--28}, + year={1966}, + publisher={Sage Publications Sage CA: Thousand Oaks, CA} +} + +@article{aumann1987correlated, + title={Correlated equilibrium as an expression of Bayesian rationality}, + author={Aumann, Robert J}, + journal={Econometrica: Journal of the Econometric Society}, + pages={1--18}, + year={1987}, + publisher={JSTOR} +} + +@book{zhang2018resource, + title={Resource Allocation in Unlicensed Long Term Evolution HetNets}, + author={Zhang, Huaqing and Khairy, Sami and Cai, Lin X and Han, Zhu}, + year={2018}, + publisher={Springer} +} + +@article{bjorn1985econometric, + title={Econometric modeling of a Stackelberg game with an application to labor force participation}, + author={Bjorn, Paul A and Vuong, Quang H}, + year={1985}, + publisher={California Institute of Technology} +} +@article{vincent1974learning, + title={Learning the optimal strategy in a zero-sum game}, + author={VINCENT, P}, + journal={Econometrica}, + volume={42}, + number={5}, + pages={885--891}, + year={1974} +} + + +@misc{vinyals2017starcraft, + title={StarCraft II: A New Challenge for Reinforcement Learning}, + author={Oriol Vinyals and Timo Ewalds and Sergey Bartunov and Petko Georgiev and Alexander Sasha Vezhnevets and Michelle Yeo and Alireza Makhzani and Heinrich Küttler and John Agapiou and Julian Schrittwieser and John Quan and Stephen Gaffney and Stig Petersen and Karen Simonyan and Tom Schaul and Hado van Hasselt and David Silver and Timothy Lillicrap and Kevin Calderone and Paul Keet and Anthony Brunasso and David Lawrence and Anders Ekermo and Jacob Repp and Rodney Tsing}, + year={2017}, + eprint={1708.04782}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@article{horgan2018distributed, + title={Distributed prioritized experience replay}, + author={Horgan, Dan and Quan, John and Budden, David and Barth-Maron, Gabriel and Hessel, Matteo and Van Hasselt, Hado and Silver, David}, + journal={arXiv preprint arXiv:1803.00933}, + year={2018} +} + +@inproceedings{mnih2016asynchronous, + title={Asynchronous methods for deep reinforcement learning}, + author={Mnih, Volodymyr and Badia, Adria Puigdomenech and Mirza, Mehdi and Graves, Alex and Lillicrap, Timothy and Harley, Tim and Silver, David and Kavukcuoglu, Koray}, + booktitle={International conference on machine learning}, + pages={1928--1937}, + year={2016}, + organization={PMLR} +} + + + + +@article{espeholt2019seed, + title={Seed rl: Scalable and efficient deep-rl with accelerated central inference}, + author={Espeholt, Lasse and Marinier, Rapha{\"e}l and Stanczyk, Piotr and Wang, Ke and Michalski, Marcin}, + journal={arXiv preprint arXiv:1910.06591}, + year={2019} +} + + +@misc{palanisamy2019multiagent, + title={Multi-Agent Connected Autonomous Driving using Deep Reinforcement Learning}, + author={Praveen Palanisamy}, + year={2019}, + eprint={1911.04175}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@misc{kurach2019google, + title={Google Research Football: A Novel Reinforcement Learning Environment}, + author={Karol Kurach and Anton Raichuk and Piotr Stańczyk and Michał Zając and Olivier Bachem and Lasse Espeholt and Carlos Riquelme and Damien Vincent and Marcin Michalski and Olivier Bousquet and Sylvain Gelly}, + year={2019}, + eprint={1907.11180}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@inproceedings{moritz2018ray, + title={Ray: A distributed framework for emerging $\{$AI$\}$ applications}, + author={Moritz, Philipp and Nishihara, Robert and Wang, Stephanie and Tumanov, Alexey and Liaw, Richard and Liang, Eric and Elibol, Melih and Yang, Zongheng and Paul, William and Jordan, Michael I and others}, + booktitle={13th $\{$USENIX$\}$ Symposium on Operating Systems Design and Implementation ($\{$OSDI$\}$ 18)}, + pages={561--577}, + year={2018} +} + + +% Chapter parallel computing + +@conference{babaeizadeh2017ga3c, + title={Reinforcement Learning thorugh Asynchronous Advantage Actor-Critic on a GPU}, + author={Babaeizadeh, Mohammad and Frosio, Iuri and Tyree, Stephen and Clemons, Jason and Kautz, Jan}, + booktitle={ICLR}, + biurl={https://openreview.net/forum?id=r1VGvBcxl}, + year={2017} +} + +@article{vinyals2019grandmaster, + title={Grandmaster level in StarCraft II using multi-agent reinforcement learning}, + author={Vinyals, Oriol and Babuschkin, Igor and Czarnecki, Wojciech M and Mathieu, Micha{\"e}l and Dudzik, Andrew and Chung, Junyoung and Choi, David H and Powell, Richard and Ewalds, Timo and Georgiev, Petko and others}, + journal={Nature}, + volume={575}, + number={7782}, + pages={350--354}, + year={2019}, + publisher={Nature Publishing Group} +} + + +@article{han2020tstarbot, + title={Tstarbot-x: An open-sourced and comprehensive study for efficient league training in starcraft ii full game}, + author={Han, Lei and Xiong, Jiechao and Sun, Peng and Sun, Xinghai and Fang, Meng and Guo, Qingwei and Chen, Qiaobo and Shi, Tengfei and Yu, Hongsheng and Wu, Xipeng and others}, + journal={arXiv preprint arXiv:2011.13729}, + year={2020} +} + +@inproceedings{wang2021scc, + title={SCC: an efficient deep reinforcement learning agent mastering the game of StarCraft II}, + author={Wang, Xiangjun and Song, Junxiao and Qi, Penghui and Peng, Peng and Tang, Zhenkun and Zhang, Wei and Li, Weimin and Pi, Xiongjun and He, Jujie and Gao, Chao and others}, + booktitle={International Conference on Machine Learning}, + pages={10905--10915}, + year={2021}, + organization={PMLR} +} + + +@article{berner2019dota, + title={Dota 2 with large scale deep reinforcement learning}, + author={Berner, Christopher and Brockman, Greg and Chan, Brooke and Cheung, Vicki and D{\k{e}}biak, Przemys{\l}aw and Dennison, Christy and Farhi, David and Fischer, Quirin and Hashme, Shariq and Hesse, Chris and others}, + journal={arXiv preprint arXiv:1912.06680}, + year={2019} +} + +@article{hoffman2020acme, + title={Acme: A research framework for distributed reinforcement learning}, + author={Hoffman, Matt and Shahriari, Bobak and Aslanides, John and Barth-Maron, Gabriel and Behbahani, Feryal and Norman, Tamara and Abdolmaleki, Abbas and Cassirer, Albin and Yang, Fan and Baumli, Kate and others}, + journal={arXiv preprint arXiv:2006.00979}, + year={2020} +} + + +@article{cassirer2021reverb, + title={Reverb: A Framework For Experience Replay}, + author={Cassirer, Albin and Barth-Maron, Gabriel and Brevdo, Eugene and Ramos, Sabela and Boyd, Toby and Sottiaux, Thibault and Kroiss, Manuel}, + journal={arXiv preprint arXiv:2102.04736}, + year={2021} +} + + +@inproceedings{espeholt2018impala, + title={Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures}, + author={Espeholt, Lasse and Soyer, Hubert and Munos, Remi and Simonyan, Karen and Mnih, Vlad and Ward, Tom and Doron, Yotam and Firoiu, Vlad and Harley, Tim and Dunning, Iain and others}, + booktitle={International Conference on Machine Learning}, + pages={1407--1416}, + year={2018}, + organization={PMLR} +} + + +@article{liang2017ray, + title={Ray rllib: A composable and scalable reinforcement learning library}, + author={Liang, Eric and Liaw, Richard and Nishihara, Robert and Moritz, Philipp and Fox, Roy and Gonzalez, Joseph and Goldberg, Ken and Stoica, Ion}, + journal={arXiv preprint arXiv:1712.09381}, + pages={85}, + year={2017} +} + + + +@misc{horgan2018distributed, + title={Distributed Prioritized Experience Replay}, + author={Dan Horgan and John Quan and David Budden and Gabriel Barth-Maron and Matteo Hessel and Hado van Hasselt and David Silver}, + year={2018}, + eprint={1803.00933}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@misc{gruslys2017reactor, + title={The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning}, + author={Audrunas Gruslys and Will Dabney and Mohammad Gheshlaghi Azar and Bilal Piot and Marc Bellemare and Remi Munos}, + year={2017}, + eprint={1704.04651}, + archivePrefix={arXiv}, + primaryClass={cs.AI} +} + +@misc{nair2015massively, + title={Massively Parallel Methods for Deep Reinforcement Learning}, + author={Arun Nair and Praveen Srinivasan and Sam Blackwell and Cagdas Alcicek and Rory Fearon and Alessandro De Maria and Vedavyas Panneershelvam and Mustafa Suleyman and Charles Beattie and Stig Petersen and Shane Legg and Volodymyr Mnih and Koray Kavukcuoglu and David Silver}, + year={2015}, + eprint={1507.04296}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@misc{openai2019dota, + title={Dota 2 with Large Scale Deep Reinforcement Learning}, + author={OpenAI and : and Christopher Berner and Greg Brockman and Brooke Chan and Vicki Cheung and Przemysław Dębiak and Christy Dennison and David Farhi and Quirin Fischer and Shariq Hashme and Chris Hesse and Rafal Józefowicz and Scott Gray and Catherine Olsson and Jakub Pachocki and Michael Petrov and Henrique Pondé de Oliveira Pinto and Jonathan Raiman and Tim Salimans and Jeremy Schlatter and Jonas Schneider and Szymon Sidor and Ilya Sutskever and Jie Tang and Filip Wolski and Susan Zhang}, + year={2019}, + eprint={1912.06680}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + + + +@article{Browne2012A, + title={A survey of monte carlo tree search methods}, + author={Browne, Cameron B. and Powley, Edward and Whitehouse, Daniel and Lucas, Simon M. and Colton, Simon}, + journal={IEEE Transactions on Computational Intelligence \& Ai in Games}, + volume={4}, + number={1}, + pages={1-43}, + year={2012}, + publisher={IEEE}, +} + +@misc{ WikipediaCGT, + author = { Wikipedia }, + title = { Combinatorial game theory }, + note = {Accessed December 20, 2019}, + year = {2019}, + howpublished = {\url{https://en.wikipedia.org/wiki/Combinatorial_game_theory}} +} + +@misc{ WikipediaPF, + author = { Wikipedia }, + title = { Perfect information }, + note = {Accessed January 8, 2020}, + year = {2020}, + howpublished = {\url{https://en.wikipedia.org/wiki/Perfect_information}} +} + +@misc{ WikipediaGomoku, + author = { Wikipedia }, + title = { Gomoku }, + note = {Accessed January 8, 2020}, + year = {2020}, + howpublished = {\url{https://en.wikipedia.org/wiki/Gomoku}} +} + +@article{Auer2002, + title={Finite-time analysis of the multiarmed bandit problem}, + author={Auer, Peter and Cesa-Bianchi, Nicolo and Fischer, Paul}, + journal={Machine learning}, + volume={47}, + number={2-3}, + pages={235--256}, + year={2002}, + publisher={Springer} +} + + +@inproceedings{He2015Deep, + title={Deep residual learning for image recognition}, + author={He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian}, + booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition}, + pages={770--778}, + year={2016} +} + +@article{silver2017mastering, + title={Mastering chess and shogi by self-play with a general reinforcement learning algorithm}, + author={Silver, David and Hubert, Thomas and Schrittwieser, Julian and Antonoglou, Ioannis and Lai, Matthew and Guez, Arthur and Lanctot, Marc and Sifre, Laurent and Kumaran, Dharshan and Graepel, Thore and others}, + journal={arXiv preprint arXiv:1712.01815}, + year={2017} +} + +@book{albert2007lessons, + title={Lessons in play: an introduction to combinatorial game theory}, + author={Albert, Michael and Nowakowski, Richard and Wolfe, David}, + year={2007}, + publisher={CRC Press} +} + + +@article{hsu1999ibm, + title={IBM's deep blue chess grandmaster chips}, + author={Hsu, Feng-hsiung}, + journal={IEEE Micro}, + volume={19}, + number={2}, + pages={70--81}, + year={1999}, + publisher={IEEE} +} + +@inproceedings{silver2014deterministic, + title={Deterministic Policy Gradient Algorithms}, + author={Silver, David and Lever, Guy and Heess, Nicolas and Degris, Thomas and Wierstra, Daan and Riedmiller, Martin}, + booktitle={International Conference on Machine Learning}, + pages={387--395}, + year={2014} +} + +@article{uhlenbeck1930theory, + title={On the theory of the Brownian motion}, + author={Uhlenbeck, George E and Ornstein, Leonard S}, + journal={Physical review}, + volume={36}, + number={5}, + pages={823}, + year={1930}, + publisher={APS} +} + +@article{it1965diffusion, + title={Diffusion processes and their sample paths}, + author={It, Kiyosi and McKean, HP}, + journal={Die Grundlehren der math. Wissenschaften}, + volume={125}, + year={1965} +} + +@article{polyak1964some, + title={Some methods of speeding up the convergence of iteration methods}, + author={Polyak, Boris T}, + journal={USSR Computational Mathematics and Mathematical Physics}, + volume={4}, + number={5}, + pages={1--17}, + year={1964}, + publisher={Elsevier} +} + +@inproceedings{munos2016safe, + title={Safe and efficient off-policy reinforcement learning}, + author={Munos, R{\'e}mi and Stepleton, Tom and Harutyunyan, Anna and Bellemare, Marc}, + booktitle={Advances in Neural Information Processing Systems}, + pages={1054--1062}, + year={2016} +} + +@article{Silver2016Mastering, + title={Mastering the game of Go with deep neural networks and tree search}, + author={Silver, David and Huang, Aja and Maddison, Chris J. and Guez, Arthur and Hassabis, Demis}, + journal={Nature}, + volume={529}, + number={7587}, + pages={484-489}, + year={2016}, +} + +@article{MuthooA, + title={A Course in Game Theory.}, + author={Muthoo, Abhinay and Osborne, Martin J and Rubinstein, Ariel}, + journal={Economica}, + volume={63}, + number={249}, + pages={164-165}, + year={1996}, +} + +@misc{schrittwieser2019mastering, + title={Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model}, + author={Julian Schrittwieser and Ioannis Antonoglou and Thomas Hubert and Karen Simonyan and Laurent Sifre and Simon Schmitt and Arthur Guez and Edward Lockhart and Demis Hassabis and Thore Graepel and Timothy Lillicrap and David Silver}, + year={2019}, + eprint={1911.08265}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + + +@misc{tensorlayerurl, +url={https://github.com/tensorlayer/tensorlayer} +} + +@inproceedings{MLSYS2021_ec895663, + author = {Jiang, Wenqi and He, Zhenhao and Zhang, Shuai and Preu\ss er, Thomas B. and Zeng, Kai and Feng, Liang and Zhang, Jiansong and Liu, Tongxuan and Li , Yong and Zhou, Jingren and Zhang, Ce and Alonso, Gustavo}, + booktitle = {Proceedings of Machine Learning and Systems}, + editor = {A. Smola and A. Dimakis and I. Stoica}, + pages = {845--859}, + title = {MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions}, + url = {https://proceedings.mlsys.org/paper/2021/file/ec8956637a99787bd197eacd77acce5e-Paper.pdf}, + volume = {3}, + year = {2021} +} + +@inproceedings{kraken, + author = {Xie, Minhui and Ren, Kai and Lu, Youyou and Yang, Guangxu and Xu, Qingxing and Wu, Bihai and Lin, Jiazhen and Ao, Hongbo and Xu, Wanhong and Shu, Jiwu}, + title = {Kraken: Memory-Efficient Continual Learning for Large-Scale Real-Time Recommendations}, + year = {2020}, + isbn = {9781728199986}, + publisher = {IEEE Press}, + abstract = {}, + booktitle = {Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis}, + articleno = {21}, + numpages = {17}, + keywords = {systems for machine learning, continual learning, recommendation system}, + location = {Atlanta, Georgia}, + series = {SC '20} +} + +@article{DBLP:journals/corr/abs-2003-05622, + author = {Weijie Zhao and + Deping Xie and + Ronglai Jia and + Yulei Qian and + Ruiquan Ding and + Mingming Sun and + Ping Li}, + title = {Distributed Hierarchical {GPU} Parameter Server for Massive Scale + Deep Learning Ads Systems}, + journal = {CoRR}, + volume = {abs/2003.05622}, + year = {2020}, + url = {https://arxiv.org/abs/2003.05622}, + eprinttype = {arXiv}, + eprint = {2003.05622}, + timestamp = {Thu, 19 Aug 2021 08:41:35 +0200}, + biburl = {https://dblp.org/rec/journals/corr/abs-2003-05622.bib}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} + +@inproceedings{NEURIPS2020_a1d4c20b, + author = {He, Chaoyang and Annavaram, Murali and Avestimehr, Salman}, + booktitle = {Advances in Neural Information Processing Systems}, + editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin}, + pages = {14068--14080}, + publisher = {Curran Associates, Inc.}, + title = {Group Knowledge Transfer: Federated Learning of Large CNNs at the Edge}, + url = {https://proceedings.neurips.cc/paper/2020/file/a1d4c20b182ad7137ab3606f0e3fc8a4-Paper.pdf}, + volume = {33}, + year = {2020} +} + +@inproceedings{gong2020edgerec, + title={EdgeRec: Recommender System on Edge in Mobile Taobao}, + author={Gong, Yu and Jiang, Ziwen and Feng, Yufei and Hu, Binbin and Zhao, Kaiqi and Liu, Qingwen and Ou, Wenwu}, + booktitle={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management}, + pages={2477--2484}, + year={2020} +} + +@inproceedings{continuum, + author = {Tian, Huangshi and Yu, Minchen and Wang, Wei}, + title = {Continuum: A Platform for Cost-Aware, Low-Latency Continual Learning}, + year = {2018}, + isbn = {9781450360111}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + url = {https://doi.org/10.1145/3267809.3267817}, + doi = {10.1145/3267809.3267817}, + abstract = {}, + booktitle = {Proceedings of the ACM Symposium on Cloud Computing}, + pages = {26–40}, + numpages = {15}, + keywords = {Continual Learning System, Competitive Analysis, Online Algorithm}, + location = {Carlsbad, CA, USA}, + series = {SoCC '18} +} + +@inproceedings{practicallessons, + author = {He, Xinran and Pan, Junfeng and Jin, Ou and Xu, Tianbing and Liu, Bo and Xu, Tao and Shi, Yanxin and Atallah, Antoine and Herbrich, Ralf and Bowers, Stuart and Candela, Joaquin Qui\~{n}onero}, + title = {Practical Lessons from Predicting Clicks on Ads at Facebook}, + year = {2014}, + isbn = {9781450329996}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + url = {https://doi.org/10.1145/2648584.2648589}, + doi = {10.1145/2648584.2648589}, + abstract = {}, + booktitle = {Proceedings of the Eighth International Workshop on Data Mining for Online Advertising}, + pages = {1–9}, + numpages = {9}, + location = {New York, NY, USA}, + series = {ADKDD'14} +} + +@inproceedings{UnbiasedOnline, + author = {Chu, Wei and Zinkevich, Martin and Li, Lihong and Thomas, Achint and Tseng, Belle}, + title = {Unbiased Online Active Learning in Data Streams}, + year = {2011}, + isbn = {9781450308137}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + url = {https://doi.org/10.1145/2020408.2020444}, + doi = {10.1145/2020408.2020444}, + abstract = {}, + booktitle = {Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining}, + pages = {195–203}, + numpages = {9}, + keywords = {adaptive importance sampling, unbiasedness, bayesian online learning, active learning, data streaming}, + location = {San Diego, California, USA}, + series = {KDD '11} +} + +@inproceedings{MLSYS2021_979d472a, + author = {Yin, Chunxing and Acun, Bilge and Wu, Carole-Jean and Liu, Xing}, + booktitle = {Proceedings of Machine Learning and Systems}, + editor = {A. Smola and A. Dimakis and I. Stoica}, + pages = {448--462}, + title = {TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models}, + url = {https://proceedings.mlsys.org/paper/2021/file/979d472a84804b9f647bc185a877a8b5-Paper.pdf}, + volume = {3}, + year = {2021} +} + +@inproceedings{10.1145/3394486.3403059, +author = {Shi, Hao-Jun Michael and Mudigere, Dheevatsa and Naumov, Maxim and Yang, Jiyan}, +title = {Compositional Embeddings Using Complementary Partitions for Memory-Efficient Recommendation Systems}, +year = {2020}, +isbn = {9781450379984}, +publisher = {Association for Computing Machinery}, +address = {New York, NY, USA}, +url = {https://doi.org/10.1145/3394486.3403059}, +doi = {10.1145/3394486.3403059}, +abstract = {}, +booktitle = {Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining}, +pages = {165–175}, +numpages = {11}, +keywords = {model compression, recommendation systems, embeddings}, +location = {Virtual Event, CA, USA}, +series = {KDD '20} +} + +@misc{ginart2021mixed, + title={Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems}, + author={Antonio Ginart and Maxim Naumov and Dheevatsa Mudigere and Jiyan Yang and James Zou}, + year={2021}, + eprint={1909.11810}, + archivePrefix={arXiv}, + primaryClass={cs.LG} +} + +@article{lamport2001paxos, + title={Paxos made simple}, + author={Lamport, Leslie and others}, + journal={ACM Sigact News}, + volume={32}, + number={4}, + pages={18--25}, + year={2001} +} + +@inproceedings{hunt2010zookeeper, + title={ZooKeeper: Wait-free Coordination for Internet-scale Systems.}, + author={Hunt, Patrick and Konar, Mahadev and Junqueira, Flavio Paiva and Reed, Benjamin}, + booktitle={USENIX annual technical conference}, + volume={8}, + number={9}, + year={2010} +} + +@inproceedings {184040, +author = {Diego Ongaro and John Ousterhout}, +title = {In Search of an Understandable Consensus Algorithm}, +booktitle = {2014 {USENIX} Annual Technical Conference ({USENIX} {ATC} 14)}, +year = {2014}, +isbn = {978-1-931971-10-2}, +address = {Philadelphia, PA}, +pages = {305--319}, +url = {https://www.usenix.org/conference/atc14/technical-sessions/presentation/ongaro}, +publisher = {{USENIX} Association}, +month = jun, +} + +@article{yu2020weips, + title={WeiPS: a symmetric fusion model framework for large-scale online learning}, + author={Yu, Xiang and Chu, Fuping and Wu, Junqi and Huang, Bo}, + journal={arXiv preprint arXiv:2011.11983}, + year={2020} +} + +@misc{program-tensorcore, + author = {Jeremy Appleyard, Scott Yokim}, + title = {Programming Tensor Cores in CUDA 9}, + howpublished = {\url{https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/}} +} + +@article{2018Modeling, + title={Modeling Deep Learning Accelerator Enabled GPUs}, + author={ Raihan, M. A. and Goli, N. and Aamodt, T. }, + year={2018}, +} + +@book{2007Engineering, + title={Engineering a Compiler}, + author={ Cooper, Keith D. and Torczon, Linda }, + publisher={Engineering A Compiler}, + year={2007}, +} + +@article{2007Compilers, + title={Compilers: Principles, Techniques, and Tools (Rental), 2nd Edition}, + author={ Aho, A. V. and Lam, M. S. and Ullman, J. D. and Sethi, R. }, + year={2007}, +} + +@inproceedings{2004LLVM, + title={LLVM: A Compilation Framework for Lifelong Program Analysis & Transformation}, + author={ Lattner, C. and Adve, V. }, + booktitle={Code Generation and Optimization, 2004. CGO 2004. International Symposium on}, + year={2004}, +} + +@article{Richard1995A, + title={A correspondence between continuation passing style and static single assignment form}, + author={Richard and A. and Kelsey}, + journal={Acm Sigplan Notices}, + year={1995}, +} + +@article{2020MLIR, + title={MLIR: A Compiler Infrastructure for the End of Moore's Law}, + author={ Lattner, C. and Amini, M. and Bondhugula, U. and Cohen, A. and Davis, A. and Pienaar, J. and Riddle, R. and Shpeisman, T. and Vasilache, N. and Zinenko, O. }, + year={2020}, +} + +@article{1969The, + title={The Principal Type-Scheme of an Object in Combinatory Logic}, + author={ Hindley, R. }, + journal={Transactions of the American Mathematical Society}, + volume={146}, + pages={29-60}, + year={1969}, +} + +@article{1978A, + title={A theory of type polymorphism in programming}, + author={ Milner, R. }, + journal={Journal of Computer and System Sciences}, + volume={17}, + number={3}, + pages={348-375}, + year={1978}, +} + +@article{1982Principal, + title={Principal Type Schemes for Functional Programming Languages}, + author={ Damas, L. and Milner, R. }, + year={1982}, +} + +@article{2015Automatic, + title={Automatic Differentiation in Machine Learning: a Survey}, + author={ Pearlmutter, B. A. }, + journal={computer science}, + number={February}, + year={2015}, +} + +@article{van2018Automatic, + title={Automatic differentiation in ML: Where we are and where we should be going}, + author={van Merriënboer, Bart and Breuleux, Olivier and Bergeron, Arnaud and Lamblin, Pascal }, + year={2018}, +} + +@book{10.5555/1455489, +author = {Griewank, Andreas and Walther, Andrea}, +title = {Evaluating Derivatives: Principles and Techniques of Algorithmic Differentiation}, +year = {2008}, +isbn = {0898716594}, +publisher = {Society for Industrial and Applied Mathematics}, +address = {USA}, +edition = {Second}, +} + +@article{2015Numerical, + title={Numerical Analysis}, + author={ Burden, R. L. and Faires, Jdd }, + journal={Journal of the Royal Statistical Society}, + volume={71}, + number={1}, + pages={48-50}, + year={2015}, +} + +@book{2003Computer, + title={Computer Algebra Handbook: Foundations * Applications * Systems}, + author={ Grabmeier, J. and Kaltofen, E. and Weispfenning, V. }, + publisher={Computer algebra handbook : foundations, applications, systems}, + year={2003}, +} + +@inbook{10.5555/60181.60188, +author = {Corliss, George F.}, +title = {Applications of Differentiation Arithmetic}, +year = {1988}, +isbn = {0125056303}, +publisher = {Academic Press Professional, Inc.}, +address = {USA}, +booktitle = {Reliability in Computing: The Role of Interval Methods in Scientific Computing}, +pages = {127–148}, +numpages = {22} +} + +@article{2000An, + title={An introduction to automatic differentiation}, + author={ Verma, A. }, + journal={Siam Computational Differentiation Techniques Applications & Tools}, + volume={78}, + number={7}, + pages={804-807}, + year={2000}, +} + +@inproceedings{2006The, + title={The Data-Flow Equations of Checkpointing in Reverse Automatic Differentiation}, + author={ Dauvergne, B. and L Hascoët}, + booktitle={Computational Science-iccs, International Conference, Reading, Uk, May}, + year={2006}, +} + +@article{2017Divide, + title={Divide-and-Conquer Checkpointing for Arbitrary Programs with No User Annotation}, + author={ Siskind, Jeffrey Mark and Pearlmutter, Barak A. }, + journal={Optimization Methods and Software}, + volume={33}, + number={4-6}, + year={2017}, +} + +@article{makoviychuk2021isaac, + title={Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning}, + author={Makoviychuk, Viktor and Wawrzyniak, Lukasz and Guo, Yunrong and Lu, Michelle and Storey, Kier and Macklin, Miles and Hoeller, David and Rudin, Nikita and Allshire, Arthur and Handa, Ankur and others}, + journal={arXiv preprint arXiv:2108.10470}, + year={2021} +} + +@article{ding2020efficient, + title={Efficient Reinforcement Learning Development with RLzoo}, + author={Ding, Zihan and Yu, Tianyang and Huang, Yanhua and Zhang, Hongming and Li, Guo and Guo, Quancheng and Mai, Luo and Dong, Hao}, + journal={arXiv preprint arXiv:2009.08644}, + year={2020} +} + +@inproceedings{yu2018dynamic, + title={Dynamic control flow in large-scale machine learning}, + author={Yu, Yuan and Abadi, Mart{\'\i}n and Barham, Paul and Brevdo, Eugene and Burrows, Mike and Davis, Andy and Dean, Jeff and Ghemawat, Sanjay and Harley, Tim and Hawkins, Peter and others}, + booktitle={Proceedings of the Thirteenth EuroSys Conference}, + pages={1--15}, + year={2018} +} + +@article{paszke2019pytorch, + title={Pytorch: An imperative style, high-performance deep learning library}, + author={Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and others}, + journal={Advances in neural information processing systems}, + volume={32}, + pages={8026--8037}, + year={2019} +} + +@misc{2019MindSpore, + author = {{Huawei MindSpore}}, + title = {MindSpore. https://github.com/mindspore-ai/mindspore.}, + year = {2019} +} + +@misc{2016Paddle, + author = {{Baidu PaddlePaddle}}, + title = {PaddlePaddle. https://www.paddlepaddle.org.c.}, + year = {2016} +}@ARTICLE{2019Pytorch, + author = {{Paszke}, Adam and {Gross}, Sam and {Massa}, Francisco , and et al}, + title = "{PyTorch: An Imperative Style, High-Performance Deep Learning Library}", + journal = {NeurIPS}, + keywords = {Computer Science - Machine Learning, Computer Science - Mathematical Software, Statistics - Machine Learning}, + year = 2019, + month = dec, + adsurl = {https://ui.adsabs.harvard.edu/abs/2019arXiv191201703P}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + +@misc{2015Keras, + author = {{Google Keras}}, + title = {Keras. https://github.com/fchollet/keras.}, + year = {2015} +} + +@ARTICLE{2016TensorFlow, + author = {{Abadi}, Mart{\'\i}n and {Barham}, Paul and {Chen}, Jianmin , and et al}, + title = "{TensorFlow: A system for large-scale machine learning}", + journal = {In Usenix OSDI}, + keywords = {Computer Science - Distributed, Parallel, and Cluster Computing, Computer Science - Artificial Intelligence}, + year = 2016, + month = may, + adsurl = {https://ui.adsabs.harvard.edu/abs/2016arXiv160508695A}, + adsnote = {Provided by the SAO/NASA Astrophysics Data System} +} + +@inproceedings{2017TensorLayer, + author = {Dong, Hao and Supratak, Akara and Mai, Luo , and et al}, + title = {TensorLayer: A Versatile Library for Efficient Deep Learning Development}, + year = {2017}, + isbn = {9781450349062}, + publisher = {Association for Computing Machinery}, + address = {New York, NY, USA}, + url = {https://doi.org/10.1145/3123266.3129391}, + doi = {10.1145/3123266.3129391}, + booktitle = {Proceedings of the 25th ACM International Conference on Multimedia}, + pages = {1201–1204}, + numpages = {4}, + keywords = {computer vision, deep learning, data management, parallel computation, natural language processing, reinforcement learning}, + location = {Mountain View, California, USA}, + series = {MM '17} +} + +@INPROCEEDINGS {2021TensorLayer3, +author = {C. Lai and J. Han and H. Dong}, +booktitle = {2021 IEEE International Conference on Multimedia & Expo Workshops (ICMEW)}, +title = {Tensorlayer 3.0: A Deep Learning Library Compatible With Multiple Backends}, +year = {2021}, +volume = {}, +issn = {}, +pages = {1-3}, +keywords = {deep learning;couplings;conferences;buildings;libraries;artificial intelligence;open source software}, +doi = {10.1109/ICMEW53276.2021.9455972}, +url = {https://doi.ieeecomputersociety.org/10.1109/ICMEW53276.2021.9455972}, +publisher = {IEEE Computer Society}, +address = {Los Alamitos, CA, USA}, +month = {jul} + +@ARTICLE{2020tkde_li, + author={Li, Xiao-Hui and Cao, Caleb Chen and Shi, Yuhan and Bai, Wei and Gao, Han and Qiu, Luyu and Wang, Cong and Gao, Yuanyuan and Zhang, Shenjia and Xue, Xun and Chen, Lei}, + journal={IEEE Transactions on Knowledge and Data Engineering}, + title={A Survey of Data-driven and Knowledge-aware eXplainable AI}, + year={2020}, + volume={}, + number={}, + pages={1-1}, + doi={10.1109/TKDE.2020.2983930}} + +@inproceedings{erhan2009visualizing, + title = {Visualizing {Higher}-{Layer} {Features} of a {Deep} {Network}}, + author = {Erhan, Dumitru and Bengio, Yoshua and Courville, Aaron C. and Vincent, Pascal}, + year = {2009}, + keywords = {Activation function, anatomical layer, Architecture as Topic, Convolutional neural network, Expectation–maximization algorithm, Experiment, High-level programming language, Interpretation Process, Noise reduction, Sampling (signal processing), Supervised learning} +} + +@InProceedings{kim2017interpretability, + title = {Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors ({TCAV})}, + author = {Kim, Been and Wattenberg, Martin and Gilmer, Justin and Cai, Carrie and Wexler, James and Viegas, Fernanda and sayres, Rory}, + booktitle = {Proceedings of the 35th International Conference on Machine Learning}, + pages = {2668--2677}, + year = {2018}, + volume = {80}, + publisher = {PMLR} +} + +@article{riedl2019human, + title={Human-centered artificial intelligence and machine learning}, + author={Riedl, Mark O.}, + journal={Human Behavior and Emerging Technologies}, + volume={1}, + number={1}, + pages={33--36}, + year={2019}, + publisher={Wiley Online Library} +} + +@article{2017NVIDIA, + title={NVIDIA Tesla V100 GPU Architecture: The World's Most Advanced Datacenter GPU. http://www.nvidia.com/object/volta-architecture-whitepaper.html.}, + author={NVIDIA}, + year={2017}, + month={Aug} +} + +@article{spuler1994compiler, + title={Compiler detection of function call side effects}, + author={Spuler, David A and Sajeev, A Sayed Muhammed}, + journal={Informatica}, + volume={18}, + number={2}, + pages={219--227}, + year={1994}, + publisher={Citeseer} +} + +@article{2010C, + title={C++ lambda expressions and closures}, + author={ Jaervi, Jaakko and Freeman, J. }, + journal={Science of Computer Programming}, + volume={75}, + number={9}, + pages={762-772}, + year={2010}, +} + +@article{DBLP:journals/corr/abs-2104-05158, + author = {Dheevatsa Mudigere and + Yuchen Hao and + Jianyu Huang and + Andrew Tulloch and + Srinivas Sridharan and + Xing Liu and + Mustafa Ozdal and + Jade Nie and + Jongsoo Park and + Liang Luo and + Jie Amy Yang and + Leon Gao and + Dmytro Ivchenko and + Aarti Basant and + Yuxi Hu and + Jiyan Yang and + Ehsan K. Ardestani and + Xiaodong Wang and + Rakesh Komuravelli and + Ching{-}Hsiang Chu and + Serhat Yilmaz and + Huayu Li and + Jiyuan Qian and + Zhuobo Feng and + Yinbin Ma and + Junjie Yang and + Ellie Wen and + Hong Li and + Lin Yang and + Chonglin Sun and + Whitney Zhao and + Dimitry Melts and + Krishna Dhulipala and + K. R. Kishore and + Tyler Graf and + Assaf Eisenman and + Kiran Kumar Matam and + Adi Gangidi and + Guoqiang Jerry Chen and + Manoj Krishnan and + Avinash Nayak and + Krishnakumar Nair and + Bharath Muthiah and + Mahmoud khorashadi and + Pallab Bhattacharya and + Petr Lapukhov and + Maxim Naumov and + Lin Qiao and + Mikhail Smelyanskiy and + Bill Jia and + Vijay Rao}, + title = {High-performance, Distributed Training of Large-scale Deep Learning + Recommendation Models}, + journal = {CoRR}, + volume = {abs/2104.05158}, + year = {2021}, + url = {https://arxiv.org/abs/2104.05158}, + eprinttype = {arXiv}, + eprint = {2104.05158}, + timestamp = {Fri, 13 Aug 2021 14:56:26 +0200}, + biburl = {https://dblp.org/rec/journals/corr/abs-2104-05158.bib}, + bibsource = {dblp computer science bibliography, https://dblp.org} +} \ No newline at end of file