This paper presents stacked attention networks (SANs) that learn to answer natural language questions from images. SANs use semantic representation of a question as query to search for the regions in an image that are related to the answer. We argue that image question answering (QA) often requires multiple steps of reasoning. Thus, we develop a multiple-layer SAN in which we query an image multiple times to infer the answer progressively. Experiments conducted on four image QA data sets demonstrate that the proposed SANs significantly outperform previous state-of-the-art approaches. The visualization of the attention layers illustrates the progress that the SAN locates the relevant visual clues that lead to the answer of the question layer-by-layer.

画像に関する質問に答える研究。CNNで画像特徴、LSTMで質問クエリ、二つ合わせて回答するのが鉄板だが、画像内の着目点をよりはっきりさせるため、画像の各領域に対するクエリのAttentionを反復して計算する(Stacked Attention)手法を提案。既存精度を大きく更新。


In this work we implement a training of a Language Model (LM), using Recurrent Neural Network (RNN) and GloVe word embeddings, introduced by Pennigton et al. in [1]. The implementation is following the general idea of training RNNs for LM tasks presented in [2], but is rather using Gated Recurrent Unit (GRU) [3] for a memory cell, and not the more commonly used LSTM [4].

RNNを使った言語モデルにword embeddingを組み込むことで性能向上をはかっている話。メモリセルにはGRU、embeddingにはGloVeを使用。n番目の単語ベクトルをn-1個の単語ベクトルから予測している。


We present a new approach to cross channel fraud detection: build graphs representing transactions from all channels and use analytics on features extracted from these graphs. Our underlying hypothesis is community based fraud detection: an account (holder) performs normal or trusted transactions within a community that is “local” to the account. We explore several notions of community based on graph properties. Our results show that properties such as shortest distance between transaction endpoints, whether they are in the same strongly connected component, whether the destination has high page rank, etc., provide excellent discriminators of fraudulent and normal transactions whereas traditional social network analysis yields poor results. Evaluation on a large dataset from a European bank shows that such methods can substantially reduce false positives in traditional fraud scoring. We show that classifiers built purely out of graph properties are very promising, with high AUC, and can complement existing fraud detection approaches.

オンラインバンキングやP2P paymentなどの複数のチャネルを統合的に扱った不正送金検出手法の提案。全チャネルをまとめてグラフを生成。グラフから仮説ベースの特徴抽出を行い、検出精度に対する実験を行っている。また、事前に特徴量を計算しておくことで、リアルタイムなFraud検出が可能としている。既存手法と比較してfalse positiveを大幅改善。


We employ deep multi-agent reinforcement learning to model the emergence of cooperation. The new notion of sequential social dilemmas allows us to model how rational agents interact, and arrive at more or less cooperative behaviours depending on the nature of the environment and the agents’ cognitive capacity. The research may enable us to better understand and control the behaviour of complex multi-agent systems such as the economy, traffic, and environmental challenges.

強化学習で、複数エージェントの場合の研究。リンゴをとるタスクで、リンゴをとる以外にビームで相手を止められるようにして実験。最初は撃たなかったが、リンゴが少なくなると敵対が発生＋複雑(賢い)なネットワークほど敵対的という結果に。また、複雑なゲームほど協調が多い傾向が出たとのこと。


Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a {\em very simple}, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a few top dominating directions from the word vectors -- that renders off-the-shelf representations {\em even stronger}. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and { text classification}) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

分散表現の精度を上げる後処理の話。分散表現の問題として、どの次元でも共通するなにがしかのベクトル量を持っていると指摘。そこでそれらを差っ引いてやることで各次元の特徴を際立たせてやろうという試み。具体的には平均と主成分をひいている。各分散表現で精度の向上を確認、特にGloveで顕著


Machine learning classifiers are known to be vulnerable to inputs maliciously constructed by adversaries to force misclassification. Such adversarial examples have been extensively studied in the context of computer vision applications. In this work, we show adversarial attacks are also effective when targeting neural network policies in reinforcement learning. Specifically, we show existing adversarial example crafting techniques can be used to significantly degrade test-time performance of trained policies. Our threat model considers adversaries capable of introducing small perturbations to the raw input of the policy. We characterize the degree of vulnerability across tasks and training algorithms, for a subclass of adversarial-example attacks in white-box and black-box settings. Regardless of the learned task or training algorithm, we observe a significant drop in performance, even with small adversarial perturbations that do not interfere with human perception. Videos are available at this http URL.

GANの話ではなくリアルにAdversarial(敵対的)な話。分類モデルで使える攻撃で、強化学習でも学習速度を遅くさせることが可能という話。目的関数の勾配にsign関数をかけたものを足すだけでOK(強化学習では選択した行動以外は勾配が入らないので学習時はsoftmaxを使用)


We present a pixel recursive super resolution model that synthesizes realistic details into images while enhancing their resolution. A low resolution image may correspond to multiple plausible high resolution images, thus modeling the super resolution process with a pixel independent conditional model often results in averaging different details--hence blurry edges. By contrast, our model is able to represent a multimodal conditional distribution by properly modeling the statistical dependencies among the high resolution image pixels, conditioned on a low resolution input. We employ a PixelCNN architecture to define a strong prior over natural images and jointly optimize this prior with a deep conditioning convolutional network. Human evaluations indicate that samples from our proposed model look more photo realistic than a strong L2 regression baseline.

ピクセル間の依存を考慮しない推定(conditioning network)に、PixelCNNを用いて計算した高解像度画像におけるピクセル間の依存(prior network)を足し合わせることで、高解像度ピクセル推定を行っている。


In this paper we propose a novel model for unconditional audio generation task that generates one audio sample at a time. We show that our model which profits from combining memory-less modules, namely autoregressive multilayer perceptron, and stateful recurrent neural networks in a hierarchical structure is de facto powerful to capture the underlying sources of variations in temporal domain for very long time on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.

音楽生成についての研究。長い時系列上での関係を捉えるために、RNNを階層状に積んで上の方ほど長い間隔の依存をとらえるのを担当するような構成を構築(最下層の出力は通常のNN)。音声合成・音楽のデータで検証しWaveNet(CNN)をうわまった、という結果。


We introduce a new model, the Recurrent Entity Network (EntNet). It is equipped with a dynamic long-term memory which allows it to maintain and update a rep- resentation of the state of the world as it receives new data. For language under- standing tasks, it can reason on-the-fly as it reads text, not just when it is required to answer a question or respond as is the case for a Memory Network (Sukhbaatar et al., 2015). Like a Neural Turing Machine or Differentiable Neural Computer (Graves et al., 2014; 2016) it maintains a fixed size memory and can learn to perform location and content-based read and write operations. However, unlike those models it has a simple parallel architecture in which several memory loca- tions can be updated simultaneously. The EntNet sets a new state-of-the-art on the bAbI tasks, and is the first method to solve all the tasks in the 10k training examples setting. We also demonstrate that it can solve a reasoning task which requires a large number of supporting facts, which other methods are not able to solve, and can generalize past its training horizon. It can also be practically used on large scale datasets such as Children’s Book Test, where it obtains competitive performance, reading the story in a single pass.

Facebookの文章読解タスクの研究。状況を記憶させるためのメモリユニットを組み合わせたネットワーク(Recurrent Entity Network)を提案。入力に対し更新を行う際key vectorを用いどのメモリに書き込むかを含め学習する。bAbIタスクで完全試合を達成。


Despite the breakthroughs in accuracy and speed of single image super-resolution using faster and deeper convolutional neural networks, one central problem remains largely unsolved: how do we recover the finer texture details when we super-resolve at large upscaling factors? The behavior of optimization-based super-resolution methods is principally driven by the choice of the objective function. Recent work has largely focused on minimizing the mean squared reconstruction error. The resulting estimates have high peak signal-to-noise ratios, but they are often lacking high-frequency details and are perceptually unsatisfying in the sense that they fail to match the fidelity expected at the higher resolution. In this paper, we present SRGAN, a generative adversarial network (GAN) for image super-resolution (SR). To our knowledge, it is the first framework capable of inferring photo-realistic natural images for 4x upscaling factors. To achieve this, we propose a perceptual loss function which consists of an adversarial loss and a content loss. The adversarial loss pushes our solution to the natural image manifold using a discriminator network that is trained to differentiate between the super-resolved images and original photo-realistic images. In addition, we use a content loss motivated by perceptual similarity instead of similarity in pixel space. Our deep residual network is able to recover photo-realistic textures from heavily downsampled images on public benchmarks. An extensive mean-opinion-score (MOS) test shows hugely significant gains in perceptual quality using SRGAN. The MOS scores obtained with SRGAN are closer to those of the original high-resolution images than to those obtained with any state-of-the-art method.

低解像度から高解像度に復元する研究。今までは本物の画像との距離(MSE)を使用していたが、これだと「感覚的」な近さと一致しないことが多かった。そこでPerceptual lossという特徴マップ間の誤差を導入(学習済みVGGを使用)。5段階の主観評価で既存モデルより1スコアUP


Deep neural networks (DNN) have revolutionized the field of natural language processing (NLP). Convolutional neural network (CNN) and recurrent neural network (RNN), the two main types of DNN architectures, are widely explored to handle various NLP tasks. CNN is supposed to be good at extracting position-invariant features and RNN at modeling units in sequence. The state of the art on many NLP tasks often switches due to the battle between CNNs and RNNs. This work is the first systematic comparison of CNN and RNN on a wide range of representative NLP tasks, aiming to give basic guidance for DNN selection.

最近のNLPではCNNかRNNがよく使われているがそのシステマティックな比較は行われてこなかった。そこでこの論文ではCNNとRNNの性能比較を行っている。具体的にはCNN、GRU、LSTMを7つのタスク(Sentiment Classification, Relation Classification, Textual Entailment, Answer Selection, Question Relation Match, Path Query Answering, POS)について評価を行っている。評価した結果、キーフレーズの認識が重要なタスク(Sentiment Detection, Question Answer Matching)以外のタスクについてはRNNの性能が上回った。


Word representations have proven useful for many NLP tasks, e.g., Brown clusters as features in dependency parsing (Koo et al., 2008). In this paper, we investigate the use of continuous word representations as features for dependency parsing. We compare several popular embeddings to Brown clusters, via multiple types of features, in both news and web domains. We find that all embeddings yield significant parsing gains, including some recent ones that can be trained in a fraction of the time of others. Explicitly tailoring the representations for the task leads to further improvements. Moreover, an ensemble of all representations achieves the best results, suggesting their complementarity.

係り受け解析の特徴としてword embeddingを使用した話。先行研究と比べてすごいのは、単語表現をタスクに合わせて調整したり、複数のアルゴリズムで得られた単語表現を組み合わせることで性能を向上させられた点。PTBとEnglish Web treebankをデータセットとして検証した結果、Ensembleした表現を使うと一番良い結果になった。


Short texts usually encounter data sparsity and ambiguity problems in representations for their lack of context. In this paper, we propose a novel method to model short texts based on semantic clustering and convolutional neural network. Particularly, we first discover semantic cliques in embedding spaces by a fast clustering algorithm. Then, multi-scale semantic units are detected under the supervision of semantic cliques, which introduce useful external knowledge for short texts. These meaningful semantic units are combined and fed into convolutional layer, followed by max-pooling operation. Experimental results on two open benchmarks validate the effectiveness of the proposed method.

短いテキストは文脈情報の不足やデータのスパース性の問題がある。これら問題に対処するためにクラスタリングとCNNを用いてテキストをモデル化する話。具体的にはembedding空間をクラスタリングすることでsemanticクリークを作成し、そこから得られたsemantic unitをCNNに入力する。文書分類タスクでstate-of-the-artな手法と比較した結果、提案手法の有効性を示すことができた。


Data sparsity is a large problem in natural language processing that refers to the fact that language is a system of rare events, so varied and complex, that even using an extremely large corpus, we can never accurately model all possible strings of words. This paper examines the use of skip-grams (a technique where by n-grams are still stored to model language, but they allow for tokens to be skipped) to overcome the data sparsity problem. We analyze this by computing all possible skip-grams in a training corpus and measure how many adjacent (standard) n-grams these cover in test documents. We examine skip-gram modelling using one to four skips with various amount of training data and test against similar documents as well as documents generated from a machine translation system. In this paper we also determine the amount of extra training data required to achieve skip-gram coverage using standard adjacent tri-grams.

言語におけるデータのスパース性の問題に対処するためにSkip-gramを使ってみた話。一般的なbigramやtrigramに比べて、skip-bigramやskip-trigramを使うことでカバレッジを向上させることができる。実際にカバレッジを比較したところ、データ数を増やすよりskip-gramを使ったほうがカバレッジ向上の役に立っている。


Detecting hypernymy relations is a key task in NLP, which is addressed in the literature using two complementary approaches. Distributional methods, whose supervised variants are the current best performers, and path-based methods, which received less research attention. We suggest an improved path-based algorithm, in which the dependency paths are encoded using a recurrent neural network, that achieves results comparable to distributional methods. We then extend the approach to integrate both path-based and distributional signals, significantly improving upon the state-of-the-art on this task.

RNNを用いて上位語の検知を行う話。具体的には依存関係のパスをLSTMsを使ってエンコードしてそれを分類している。評価した結果、従来よく行われているDistributionalの方法に匹敵する性能を示した。また、Distributionalな手法と組み合わせることでF1で14ポイントの向上が見られた。


We introduce an extension to the bag-ofwords model for learning words representations that take into account both syntactic and semantic properties within language. This is done by employing an attention model that finds within the contextual words, the words that are relevant for each prediction. The general intuition of our model is that some words are only relevant for predicting local context (e.g. function words), while other words are more suited for determining global context, such as the topic of the document. Experiments performed on both semantically and syntactically oriented tasks show gains using our model over the existing bag of words model. Furthermore, compared to other more sophisticated models, our model scales better as we increase the size of the context of the model.

「ある単語の予測は文脈内のある単語に大きく依存している」という仮説を考慮して単語埋め込み表現を獲得するために、CBOWにAttentionを導入した話。POS Induction(教師なしの品詞タグ付け)、品詞タグ付け、評判分析で評価したところ、POS Inductionに対しては既存の手法(CBOW, Skip-ngram, SSkip-ngram)と比較して良い結果であった。その他タスクでもそこそこの性能を示した。


For artificial general intelligence (AGI) it would be efficient if multiple users trained the same giant neural network, permitting parameter reuse, without catastrophic forgetting. PathNet is a first step in this direction. It is a neural network algorithm that uses agents embedded in the neural network whose task is to discover which parts of the network to re-use for new tasks. Agents are pathways (views) through the network which determine the subset of parameters that are used and updated by the forwards and backwards passes of the backpropogation algorithm. During learning, a tournament selection genetic algorithm is used to select pathways through the neural network for replication and mutation. Pathway fitness is the performance of that pathway measured according to a cost function. We demonstrate successful transfer learning; fixing the parameters along a path learned on task A and re-evolving a new population of paths for task B, allows task B to be learned faster than it could be learned from scratch or after fine-tuning. Paths evolved on task B re-use parts of the optimal path evolved on task A. Positive transfer was demonstrated for binary MNIST, CIFAR, and SVHN supervised learning classification tasks, and a set of Atari and Labyrinth reinforcement learning tasks, suggesting PathNets have general applicability for neural network training. Finally, PathNet also significantly improves the robustness to hyperparameter choices of a parallel asynchronous reinforcement learning algorithm (A3C).

同構造のネットワーク同士で、互いの学習結果を壊さず相手の学習結果を取り込むことを目指した研究。学習と同時にモジュール(畳込層など)を結ぶパスを遺伝的アルゴリズムで進化させていき、学習が完了したら重みを固定し次のタスクに入る。これで効果的な転移が可能なことを確認(画像&強化学習)。


Deep artificial neural networks have made remarkable progress in different tasks in the field of computer vision. However, the empirical analysis of these models and investigation of their failure cases has received attention recently. In this work, we show that deep learning models cannot generalize to atypical images that are substantially different from training images. This is in contrast to the superior generalization ability of the visual system in the human brain. We focus on Convolutional Neural Networks (CNN) as the state-of-the-art models in object recognition and classification; investigate this problem in more detail, and hypothesize that training CNN models suffer from unstructured loss minimization. We propose computational models to improve the generalization capacity of CNNs by considering how typical a training image looks like. By conducting an extensive set of experiments we show that involving a typicality measure can improve the classification results on a new set of images by a large margin. More importantly, this significant improvement is achieved without fine-tuning the CNN model on the target image set.

分類タスクの学習に対して全ての入力データでlossの重みが同一で良いかを検証し、重み付けの手法を提案。同一クラス内のデータに対してTypical, Atypicalなデータを区別するためにTypical Scoreを算出し、lossのweightとして利用。Typical Scoreはクラス内の重心（平均値など）っぽさ（近さ）。weightは非線形なものも試している。


In this paper, drawing intuition from the Turing test, we propose using adversarial training for open-domain dialogue generation: the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances. We cast the task as a reinforcement learning (RL) problem where we jointly train two systems, a generative model to produce response sequences, and a discriminator---analagous to the human evaluator in the Turing test--- to distinguish between the human-generated dialogues and the machine-generated ones. The outputs from the discriminator are then used as rewards for the generative model, pushing the system to generate dialogues that mostly resemble human dialogues. In addition to adversarial training we describe a model for adversarial {\em evaluation} that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls. Experimental results on several metrics, including adversarial evaluation, demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines.

対話モデルについて、チューリングテストのように「生成した発話が人間と区別できないか」を評価する、つまりGANと同じ仕組みの導入を提案。Seq2Seqを基本とし、騙せたかどうかで報酬を得る強化学習の仕組みを導入(生成単語ごとに判定を実施)。既存のSeq2Seqモデルより優秀な結果。


Oriental ink painting, called Sumi-e, is one of the most appealing painting styles that has attracted artists around the world. Major challenges in computer-based Sumi-e simulation are to abstract complex scene information and draw smooth and natural brush strokes. To automatically find such strokes, we propose to model the brush as a reinforcement learning agent, and learn desired brush-trajectories by maximizing the sum of rewards in the policy search framework. We also provide elaborate design of actions, states, and rewards tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through simulated Sumi-e experiments.

墨絵における筆の運びを強化学習で学習させる話。筆の動き・止め・跳ね・回転をアクションとし、筆の動きの滑らかさを報酬として学習をしている。状態はグロバール上の位置と、それを基に計算するストロークの中における相対情報(Figure2参照)の双方を扱っている(計算に使うのは相対のみ)。


Task-oriented dialogue focuses on conversational agents that participate in user-initiated dialogues on domain-specific topics. In contrast to chatbots, which simply seek to sustain open-ended meaningful discourse, existing task-oriented agents usually explicitly model user intent and belief states. This paper examines bypassing such an explicit representation by depending on a latent neural embedding of state and learning selective attention to dialogue history together with copying to incorporate relevant prior context. We complement recent work by showing the effectiveness of simple sequence-to-sequence neural architectures with a copy mechanism. Our model outperforms more complex memory-augmented models by 7% in per-response generation and is on par with the current state-of-the-art on DSTC2.

Seq2Seqでタスク指向対話を行う話。単純に出力(システム発話)を予測させるだけでなく、Attentionが最も高い入力ベクトルを予測させる(入力からコピーして教師ベクトルにする)。さらに知識ベースに入っている語か否かの情報を加えてそこそこの精度なので、結果はちょっと微妙。


We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the corresponding optimization problem is sound, and provide extensive theoretical work highlighting the deep connections to other distances between distributions.

VAEやGANなどの生成系のタスクでは、「真の分布」との距離の最小化を目的にしている。つまり「距離」の定義はモデルの精度の大きな要素で、GANではこの自由度が高い反面学習が安定しない要因になっていた。そこでWasserstein距離を使うと勾配が消失せず学習が安定したという話。


State-of-the-art named entity recognition systems rely heavily on hand-crafted features and domain-specific knowledge in order to learn effectively from the small, supervised training corpora that are available. In this paper, we introduce two new neural architectures---one based on bidirectional LSTMs and conditional random fields, and the other that constructs and labels segments using a transition-based approach inspired by shift-reduce parsers. Our models rely on two sources of information about words: character-based word representations learned from the supervised corpus and unsupervised word representations learned from unannotated corpora. Our models obtain state-of-the-art performance in NER in four languages without resorting to any language-specific knowledge or resources such as gazetteers.

言語固有の知識やリソースに依存しないような固有表現認識の手法を提案した話。具体的には2つのモデル(LSTM-CRFとS-LSTM)を提案している。ラベル付きコーパスから学習した文字ベースの単語表現とラベルなしコーパスから学習した単語表現を入力とすることで、4つの言語でSOTAとなった。


This paper presents a new semi-supervised framework with convolutional neural networks (CNNs) for text categorization. Unlike the previous approaches that rely on word embeddings, our method learns embeddings of small text regions from unlabeled data for integration into a supervised CNN. The proposed scheme for embedding learning is based on the idea of two-view semi-supervised learning, which is intended to be useful for the task of interest even though the training is done on unlabeled data. Our models achieve better results than previous approaches on sentiment classification and topic classification tasks.

テキスト分類のためにCNNを使った半教師あり学習のフレームワークを提案した話。従来モデルでは事前学習済みのword embeddingを畳み込み層の入力に使っていたが、本研究では小さいテキストの領域から教師なしでembeddingを学習し、教師ありCNNにおける畳み込み層の入力の一部として使う。評価分析(IMDB, Elec)とトピック分類(RCV1)で実験したところ、先行研究より高い性能を示した。


We describe a convolutional neural network that learns feature representations for short textual posts using hashtags as a supervised signal. The proposed approach is trained on up to 5.5 billion words predicting 100,000 possible hashtags. As well as strong performance on the hashtag prediction task itself, we show that its learned representation of text (ignoring the hashtag labels) is useful for other tasks as well. To that end, we present results on a document recommendation task, where it also outperforms a number of baselines.

ハッシュタグを教師にして短いテキストの表現を学習する話。具体的にはCNNを用いて、テキストとハッシュタグのペアに対してスコアを出力し、ハッシュタグのランク付けを行う過程でテキストの表現を学習する。ハッシュタグの予測と文書推薦タスクで評価を行った結果、ベースラインの手法よりも良い結果となった。


Convolutional Neural Networks (CNNs) have recently achieved remarkably strong performance on the practically important task of sentence classification (kim 2014, kalchbrenner 2014, johnson 2014). However, these models require practitioners to specify an exact model architecture and set accompanying hyperparameters, including the filter region size, regularization parameters, and so on. It is currently unknown how sensitive model performance is to changes in these configurations for the task of sentence classification. We thus conduct a sensitivity analysis of one-layer CNNs to explore the effect of architecture components on model performance; our aim is to distinguish between important and comparatively inconsequential design decisions for sentence classification. We focus on one-layer CNNs (to the exclusion of more complex models) due to their comparative simplicity and strong empirical performance, which makes it a modern standard baseline method akin to Support Vector Machine (SVMs) and logistic regression. We derive practical advice from our extensive empirical results for those interested in getting the most out of CNNs for sentence classification in real world settings.

CNNのモデルは文分類でいい結果を残しているけど、熟練者がアーキテクチャ決めたりハイパーパラメータを設定する必要がある。これらの変更がどのような結果を及ぼすのかよくわからないので、一層のCNNを使って検証した話。最後に、CNNで文分類するときにモデルのアーキテクチャやハイパーパラメータをどう設定すべきか実践的なアドバイスをしている。


We propose a unified neural network architecture and learning algorithm that can be applied to various natural language processing tasks including: part-of-speech tagging, chunking, named entity recognition, and semantic role labeling. This versatility is achieved by trying to avoid task-specific engineering and therefore disregarding a lot of prior knowledge. Instead of exploiting man-made input features carefully optimized for each task, our system learns internal representations on the basis of vast amounts of mostly unlabeled training data. This work is then used as a basis for building a freely available tagging system with good performance and minimal computational requirements.

品詞タグ付け、チャンキング、固有表現抽出、意味役割付与を学習できるニューラルネットワークを提案した話。単純に学習させるだけではベンチマークより性能が下回ったが、ラベルなしデータを用いて言語モデルの学習を事前に行うことで、質の良い単語ベクトルが性能向上に寄与することを示した。さらに各タスクを解くためのモデル間でパラメタを共有してマルチタスク学習を行うことで性能がより向上することも示した。


Convolutional neural network (CNN) is a neural network that can make use of the internal structure of data such as the 2D structure of image data. This paper studies CNN on text categorization to exploit the 1D structure (namely, word order) of text data for accurate prediction. Instead of using low-dimensional word vectors as input as is often done, we directly apply CNN to high-dimensional text data, which leads to directly learning embedding of small text regions for use in classification. In addition to a straightforward adaptation of CNN from image to text, a simple but new variation which employs bag-of-word conversion in the convolution layer is proposed. An extension to combine multiple convolution layers is also explored for higher accuracy. The experiments demonstrate the effectiveness of our approach in comparison with state-of-the-art methods.

CNNを使って語順を考慮したテキスト分類を行う話。たいていのCNNの手法では入力としてword embeddingを入力するが、この研究では高次元のone-hotベクトルをそのまま入力して、小さなテキスト領域のembeddingを学習する。評判分析(IMDB含む)とトピック分類に関する3つのデータセットでSOTAな手法と比較した結果、提案手法の有効性を示せた。


This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.

文字レベルの畳み込みニューラルネットワークをテキスト分類に使った話。シソーラスを使ってテキスト中の単語を同義語で置換することでデータを増やしている。比較は、伝統的な手法としてbow、bag-of-ngram、bag-of-means、Deep Learning手法として、単語ベースのCNN、LSTMを対象に行っている。8つのデータセットを作成してベースの手法と比較した結果、いくつかのデータセットでは有効性を示せた。


We propose a conceptually simple and lightweight framework for deep reinforcement learning that uses asynchronous gradient descent for optimization of deep neural network controllers. We present asynchronous variants of four standard reinforcement learning algorithms and show that parallel actor-learners have a stabilizing effect on training allowing all four methods to successfully train neural network controllers. The best performing method, an asynchronous variant of actor-critic, surpasses the current state-of-the-art on the Atari domain while training for half the time on a single multi-core CPU instead of a GPU. Furthermore, we show that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

並列に動作する複数エージェントのサンプルから学習する強化学習手法A3Cの提案。複数のエージェントごとに並列にサンプルを収集し勾配を評価しロスを足しあげ、その勾配を使い共有パラメータを非同期に更新する。また非同期に共有パラメータを各エージェントのパラメータとして取得する。サンプルは保持しないのでexperience replayは必要がない。DQN,DDQNなどと比較しCPUを使い短い学習時間であってもより高性能、この時点のSOTA.


Semantic Textual Similarity (STS) seeks to measure the degree of semantic equivalence between two snippets of text. Similarity is expressed on an ordinal scale that spans from semantic equivalence to complete unrelatedness. Intermediate values capture specifically defined levels of partial similarity. While prior evaluations constrained themselves to just monolingual snippets of text, the 2016 shared task includes a pilot subtask on computing semantic similarity on cross-lingual text snippets. This year’s traditional monolingual subtask involves the evaluation of English text snippets from the following four domains: Plagiarism Detection, Post-Edited Machine Translations, Question-Answering and News Article Headlines. From the questionanswering domain, we include both questionquestion and answer-answer pairs. The cross-lingual subtask provides paired SpanishEnglish text snippets drawn from the same sources as the English data as well as independently sampled news data. The English subtask attracted 43 participating teams producing 119 system submissions, while the crosslingual Spanish-English pilot subtask attracted 10 teams resulting in 26 systems.

テキストの類似性を測るタスクであるSemEval-2016 Task 1の説明をしている論文。説明内容はタスクの説明、アノテーション方法の説明、参加者が提出した結果の提示、および総括からなる。2016年はcross-lingualなテキストの類似性を含んだのが特徴。


Distributed word representations have recently been proven to be an invaluable resource for NLP. These representations are normally learned using neural networks and capture syntactic and semantic information about words. Information about word morphology and shape is normally ignored when learning word representations. However, for tasks like part-of-speech tagging, intra-word information is extremely useful, specially when dealing with morphologically rich languages. In this paper, we propose a deep neural network that learns character-level representation of words and associate them with usual word representations to perform POS tagging. Using the proposed approach, while avoiding the use of any handcrafted feature, we produce state-of-the-art POS taggers for two languages: English, with 97.32% accuracy on the Penn Treebank WSJ corpus; and Portuguese, with 97.47% accuracy on the Mac-Morpho corpus, where the latter represents an error reduction of 12.2% on the best previous known result.

品詞タグ付けをCNN(CharWNN)を使って行う話。具体的には、単語レベルと文字レベルのembeddingsを統合して単語のベクトル表現を構築し、構築したベクトルを入力することで品詞のスコアを出力するCNNを構築した。英語とポルトガル語に対するデータセット(WSJとMac-Morpho)を用いて実験した結果、SOTAな結果となった。


Sentiment analysis of short texts such as single sentences and Twitter messages is challenging because of the limited contextual information that they normally contain. Effectively solving this task requires strategies that combine the small text content with prior knowledge and use more than just bag-of-words. In this work we propose a new deep convolutional neural network that exploits from character- to sentence-level information to perform sentiment analysis of short texts. We apply our approach for two corpora of two different domains: the Stanford Sentiment Treebank (SSTb), which contains sentences from movie reviews; and the Stanford Twitter Sentiment corpus (STS), which contains Twitter messages. For the SSTb corpus, our approach achieves state-of-the-art results for single sentence sentiment prediction in both binary positive/negative classification, with 85.7% accuracy, and fine-grained classification, with 48.3% accuracy. For the STS corpus, our approach achieves a sentiment prediction accuracy of 86.4%.

映画レビューやTwitterに対する評判分析をCNN(CharSCNN)を使って行う話。具体的には、単語レベルと文字レベルのembeddingsから文のベクトル表現を構築し、構築したベクトルを入力することで評判のスコアを出力するCNNを構築した。映画レビュー(SSTb)とTwitter(STS)に対するデータセットを用いて実験した結果、SOTAな結果となった。


This paper addresses the automatic generation of a typographic font from a subset of characters. Specifically, we use a subset of a typographic font to extrapolate additional characters. Consequently, we obtain a complete font containing a number of characters sufficient for daily use. The automated generation of Japanese fonts is in high demand because a Japanese font requires over 1,000 characters. Unfortunately, professional typographers create most fonts, resulting in significant financial and time investments for font generation. The proposed method can be a great aid for font creation because designers do not need to create the majority of the characters for a new font. The proposed method uses strokes from given samples for font generation. The strokes, from which we construct characters, are extracted by exploiting a character skeleton dataset. This study makes three main contributions: a novel method of extracting strokes from characters, which is applicable to both standard fonts and their variations; a fully automated approach for constructing characters; and a selection method for sample characters. We demonstrate our proposed method by generating 2,965 characters in 47 fonts. Objective and subjective evaluations verify that the generated characters are similar to handmade characters.

日本語フォントを作るにはたくさん文字を書かないといけないので、それを楽にしようという論文。基準となる文字セット(Skeleton dataset)を用意し、それと新しいフォントの数文字について比較を行う。そこから構造(傾きなど)とストロークを抽出し、他の文字にも適用するという手法


Attention plays a critical role in human visual experience. Furthermore, it has recently been demonstrated that attention can also play an important role in the context of applying artificial neural networks to a variety of tasks from fields such as computer vision and NLP. In this work we show that, by properly defining attention for convolutional neural networks, we can actually use this type of information in order to significantly improve the performance of a student CNN network by forcing it to mimic the attention maps of a powerful teacher network. To that end, we propose several novel methods of transferring attention, showing consistent improvement across a variety of datasets and convolutional neural network architectures. Code and models for our experiments are available at this https URL

画像を認識するとき、この辺に注目するとええよ、というようにAttentionをTransferするという研究。Activationベース(lossを調整)と、Gradientベース(勾配を調整)の2種類を提案。効果は微少な感じだが、Activationベースの方が有効


We study the problem of transferring a sample in one domain to an analog sample in another domain. Given two related domains, S and T, we would like to learn a generative function G that maps an input sample from S to the domain T, such that the output of a given function f, which accepts inputs in either domains, would remain unchanged. Other than the function f, the training data is unsupervised and consist of a set of samples from each domain. The Domain Transfer Network (DTN) we present employs a compound loss function that includes a multiclass GAN loss, an f-constancy component, and a regularizing component that encourages G to map samples from T to themselves. We apply our method to visual domains including digits and face images and demonstrate its ability to generate convincing novel images of previously unseen entities, while preserving their identity.

ドメイン(スタイル)トランスファーの研究で、スタイルを変換する関数Gを画像の特徴抽出(f))とスタイル変換(g)に分割し、あたかもアナロジーを行うように(元画像＋スタイル)転化を行っている。これにより汎用性が高くなっている。メインのタスクとして、顔画像をアイコン風に変換している。


We propose an end-to-end learning framework for generating foreground object segmentations. Given a single novel image, our approach produces pixel-level masks for all "object-like" regions---even for object categories never seen during training. We formulate the task as a structured prediction problem of assigning foreground/background labels to all pixels, implemented using a deep fully convolutional network. Key to our idea is training with a mix of image-level object category examples together with relatively few images with boundary-level annotations. Our method substantially improves the state-of-the-art on foreground segmentation for ImageNet and MIT Object Discovery datasets. Furthermore, on over 1 million images, we show that it generalizes well to segment object categories unseen in the foreground maps used for training. Finally, we demonstrate how our approach benefits image retrieval and image retargeting, both of which flourish when given our high-quality foreground maps.

オブジェクト検知をピクセルレベルで行った話。前景部分の抽出、領域提案といった既存の手法と異なり、ピクセル単位で前景/背景の確率をCNNで計算する。 クラス識別のモデルを、セグメンテーションのデータセットでファインチューニングするという面白い手法をとっている。


Despite recent advances, memory-augmented deep neural networks are still limited when it comes to life-long and one-shot learning, especially in remembering rare events. We present a large-scale life-long memory module for use in deep learning. The module exploits fast nearest-neighbor algorithms for efficiency and thus scales to large memory sizes. Except for the nearest-neighbor query, the module is fully differentiable and trained end-to-end with no extra supervision. It operates in a life-long manner, i.e., without the need to reset it during training. Our memory module can be easily added to any part of a supervised neural network. To show its versatility we add it to a number of networks, from simple convolutional ones tested on image classification to deep sequence-to-sequence and recurrent-convolutional models. In all cases, the enhanced network gains the ability to remember and do life-long one-shot learning. Our module remembers training examples shown many thousands of steps in the past and it can successfully generalize from them. We set new state-of-the-art for one-shot learning on the Omniglot dataset and demonstrate, for the first time, life-long one-shot learning in recurrent neural networks on a large-scale machine translation task.

外部メモリを利用したone-shot learningの精度向上手法を提案。one-shot learningタスクではSoTA。hashing trickを使って最近傍法を効率的に実行。memory lossという独自のlossを導入している。


We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.

CNNを用いて文分類を行う話。具体的には文を単語ベクトルの列として表し、それに対してCNNを用いて特徴抽出・分類を行っている。論文では事前学習済みの単語ベクトル(Google Newsをword2vecで学習したもの)を使っている。評価分析や質問タイプ分類を含む7つのタスクで評価したところ、7つ中4つでSOTAな結果になった。


We present Tweet2Vec, a novel method for generating general-purpose vector representation of tweets. The model learns tweet embeddings using character-level CNN-LSTM encoder-decoder. We trained our model on 3 million, randomly selected English-language tweets. The model was evaluated using two methods: tweet semantic similarity and tweet sentiment categorization, outperforming the previous state-of-the-art in both tasks. The evaluations demonstrate the power of the tweet embeddings generated by our model for various tweet categorization tasks. The vector representations generated by our model are generic, and hence can be applied to a variety of tasks. Though the model presented in this paper is trained on English-language tweets, the method presented can be used to learn tweet embeddings for different languages.

文字レベルのCNN-LSTM Encoder-Decoderモデルを構築して、Tweetのembeddingsを学習する話。文字レベルで処理を行うことで、Tweetに混じるノイズに頑健になり、他言語でも適用可能となった。Semantic Relatednessと評価分類で評価した結果これまでで最高の性能を達成した。


This paper introduces WaveNet, a deep neural network for generating raw audio waveforms. The model is fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones; nonetheless we show that it can be efficiently trained on data with tens of thousands of samples per second of audio. When applied to text-to-speech, it yields state-of-the-art performance, with human listeners rating it as significantly more natural sounding than the best parametric and concatenative systems for both English and Mandarin. A single WaveNet can capture the characteristics of many different speakers with equal fidelity, and can switch between them by conditioning on the speaker identity. When trained to model music, we find that it generates novel and often highly realistic musical fragments. We also show that it can be employed as a discriminative model, returning promising results for phoneme recognition.

PixelCNNなどでうまくいっている自己回帰モデルによる生成モデルを音声の生成に適用してみた。dilationが幾何級数的に大きくなるconvolutionの列の導入により波形の直接生成に必要な大きな受容野を計算量を大きくせず実現できた。主観評価でこれまでで最高の品質を実現。音素認識にも使えて、有望な結果を出した。


Text from social media provides a set of challenges that can cause traditional NLP approaches to fail. Informal language, spelling errors, abbreviations, and special characters are all commonplace in these posts, leading to a prohibitively large vocabulary size for word-level approaches. We propose a character composition model, tweet2vec, which finds vector-space representations of whole tweets by learning complex, non-local dependencies in character sequences. The proposed model outperforms a word-level baseline at predicting user-annotated hashtags associated with the posts, doing significantly better when the input contains many out-of-vocabulary words or unusual character sequences. Our tweet2vec encoder is publicly available.

Twitterの投稿内容から投稿についているハッシュタグを予測する文字ベースのニューラルネットワーク(Bi-GRU)を構築する話。文字ベースで予測することで膨大な単語を扱う必要がない、未知語に強い、単語分割が必要ない問いった利点がある。ハッシュタグの予測性能で評価した結果、単語レベルの方法に比べて良い性能を示した。


We present a novel word level vector representation based on symmetric patterns (SPs). For this aim we automatically acquire SPs (e.g., “X and Y”) from a large corpus of plain text, and generate vectors where each coordinate represents the cooccurrence in SPs of the represented word with another word of the vocabulary. Our representation has three advantages over existing alternatives: First, being based on symmetric word relationships, it is highly suitable for word similarity prediction. Particularly, on the SimLex999 word similarity dataset, our model achieves a Spearman’s ρ score of 0.517, compared to 0.462 of the state-of-the-art word2vec model. Interestingly, our model performs exceptionally well on verbs, outperforming stateof-the-art baselines by 20.2–41.5%. Second, pattern features can be adapted to the needs of a target NLP application. For example, we show that we can easily control whether the embeddings derived from SPs deem antonym pairs (e.g. (big,small)) as similar or dissimilar, an important distinction for tasks such as word classification and sentiment analysis. Finally, we show that a simple combination of the word similarity scores generated by our method and by word2vec results in a superior predictive power over that of each individual model, scoring as high as 0.563 in Spearman’s ρ on SimLex999. This emphasizes the differences between the signals captured by each of the models.

Symmetric Pattern(SP)(たとえばX and Y)に基づく単語ベクトル表現を提案した話。手法としてはSPをコーパスから獲得し、それに基づきPPMIを用いてベクトルを生成する。単語類似度タスクで評価した結果、SimLex999ではSOTAとなった。また、動詞に対して有効であることも分かった。


Despite interest in using cross-lingual knowledge to learn word embeddings for various tasks, a systematic comparison of the possible approaches is lacking in the literature. We perform an extensive evaluation of four popular approaches of inducing cross-lingual embeddings, each requiring a different form of supervision, on four typographically different language pairs. Our evaluation setup spans four different tasks, including intrinsic evaluation on mono-lingual and cross-lingual similarity, and extrinsic evaluation on downstream semantic and syntactic applications. We show that models which require expensive cross-lingual knowledge almost always perform better, but cheaply supervised models often prove competitive on certain tasks.

多言語情報を使ってword embeddingsを得ることで性能向上することは知られていたが、手法の比較は行われてこなかった。そのため、4つの手法を比較した話(BiSkip, BiCVM, BiVCD, BiCCA)。Intrinsicなタスク(monolingualとcross-lingualでの単語類似度タスク)とExtrinsicなタスク(cross-lingualでの文書分類と係り受け解析)で評価した結果、単言語の類似度タスクではBiSkipとBiVCDは同じくらいだが、cross-lingualなタスクではBiSkipがかなり良い結果となることを示し、対照的にSyntacticなタスクではBiCCAが最も良い結果となった。


We propose Edward, a Turing-complete probabilistic programming language. Edward defines two compositional representations---random variables and inference. By treating inference as a first class citizen, on a par with modeling, we show that probabilistic programming can be as flexible and computationally efficient as traditional deep learning. For flexibility, Edward makes it easy to fit the same model using a variety of composable inference methods, ranging from point estimation to variational inference to MCMC. In addition, Edward can reuse the modeling representation as part of inference, facilitating the design of rich variational models and generative adversarial networks. For efficiency, Edward is integrated into TensorFlow, providing significant speedups over existing probabilistic systems. For example, we show on a benchmark logistic regression task that Edward is at least 35x faster than Stan and 6x faster than PyMC3. Further, Edward incurs no runtime overhead: it is as fast as handwritten TensorFlow.

確率モデルを記述するためのフレームワークEdwardの論文。確率変数をつないでグラフ構造を構築するように記載でき、TensorFlowのグラフと統合し爆速で動作可能。VAEやGANなど、確率モデルと統合されたようなモデルが書きやすくなる。


One characteristic that sets humans apart from modern learning-based computer vision algorithms is the ability to acquire knowledge about the world and use that knowledge to reason about the visual world. Humans can learn about the characteristics of objects and the relationships that occur between them to learn a large variety of visual concepts, often with few examples. This paper investigates the use of structured prior knowledge in the form of knowledge graphs and shows that using this knowledge improves performance on image classification. We build on recent work on end-to-end learning on graphs, introducing the Graph Search Neural Network as a way of efficiently incorporating large knowledge graphs into a vision classification pipeline. We show in a number of experiments that our method outperforms standard neural network baselines for multi-label classification.

画像認識について、人間は少ないサンプルですぐに識別ができるが、機械学習モデルはそうはいかない。人間は認識結果をもとにその知識から推論を行っているからそれができる、と仮定し、この「知識」=認識画像間の関係グラフを組み込むことで1-shotの精度を上げようという話。 MSCOCOの事前学習モデルで識別を行い、Visual Genomeというよりスパースなデータセットで精度を検証。1-shot/5-shotでVGGの1.4~5倍の精度を達成。


In this paper, we propose LexVec, a new method for generating distributed word representations that uses low-rank, weighted factorization of the Positive Point-wise Mutual Information matrix via stochastic gradient descent, employing a weighting scheme that assigns heavier penalties for errors on frequent cooccurrences while still accounting for negative co-occurrence. Evaluation on word similarity and analogy tasks shows that LexVec matches and often outperforms state-of-the-art methods on many of these tasks.

LexVecという単語埋め込みベクトル獲得方法を提案した話。具体的にはPPMI行列を分解することによって得る。単語類似度タスクとアナロジータスクで評価した結果、単語類似度タスクではいくつかの評価セットにおいてSGNSを上回ったもののアナロジータスクではSGNSやGloVeの方が良い結果となった。


Word embeddings – distributed representations of words – in deep learning are beneficial for many tasks in NLP. However, different embedding sets vary greatly in quality and characteristics of the captured information. Instead of relying on a more advanced algorithm for embedding learning, this paper proposes an ensemble approach of combining different public embedding sets with the aim of learning metaembeddings. Experiments on word similarity and analogy tasks and on part-of-speech tagging show better performance of metaembeddings compared to individual embedding sets. One advantage of metaembeddings is the increased vocabulary coverage. We release our metaembeddings publicly at http:// cistern.cis.lmu.de/meta-emb.

異なる性質を持つembedding集合を組み合わせてmeta embeddingを得る話。具体的には5つのembedding集合(HLBL, Huang, GloVe, CW, word2vec)を4つの手法(CONC, SVD, 1toN, 1toN+)で組み合わせて実験。これにより、単語類似度タスク、アナロジータスク、POSの性能が向上した。また組み合わせることでボキャブラリのカバレッジをあげられるのもメリット。


Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research. We introduce SimVerb-3500, an evaluation resource that provides human ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed verb types from the USF free-association database, providing at least three examples for every VerbNet class. This broad coverage facilitates detailed analyses of how syntactic and semantic phenomena together influence human understanding of verb meaning. Further, with significantly larger development and test sets than existing benchmarks, SimVerb-3500 enables more robust evaluation of representation learning architectures and promotes the development of methods tailored to verbs. We hope that SimVerb-3500 will enable a richer understanding of the diversity and complexity of verb semantics and guide the development of systems that can effectively represent and interpret this meaning.

動詞は文の意味を決定するのに重要な役割を占めるのに、最近の単語の意味研究ではあんまり注目されていないよねということで3500の動詞ペアの類似度を人間が評価したデータセットを提案した話。既存の単語表現学習モデルで分析したところ、低頻度で多義の動詞については非常に低い性能となることがわかった。


We introduce an exceptionally simple gated recurrent neural network (RNN) that achieves performance comparable to well-known gated architectures, such as LSTMs and GRUs, on the word-level language modeling task. We prove that our model has simple, predicable and non-chaotic dynamics. This stands in stark contrast to more standard gated architectures, whose underlying dynamical systems exhibit chaotic behavior.

GRUよりさらに単純化したRNNゲートアーキテクチャの提案。hidden stateから次の時刻のhidden stateへの重み行列を単位行列で置き換えた。言語モデルではLSTMやGRUと同等のパフォーマンスを示した。


Modern automatic speech recognition (ASR) systems need to be robust under acoustic variability arising from environmental, speaker, channel, and recording conditions. Ensuring such robustness to variability is a challenge in modern day neural network-based ASR systems, especially when all types of variability are not seen during training. We attempt to address this problem by encouraging the neural network acoustic model to learn invariant feature representations. We use ideas from recent research on image generation using Generative Adversarial Networks and domain adaptation ideas extending adversarial gradient-based training. A recent work from Ganin et al. proposes to use adversarial training for image domain adaptation by using an intermediate representation from the main target classification network to deteriorate the domain classifier performance through a separate neural network. Our work focuses on investigating neural architectures which produce representations invariant to noise conditions for ASR. We evaluate the proposed architecture on the Aurora-4 task, a popular benchmark for noise robust ASR. We show that our method generalizes better than the standard multi-condition training especially when only a few noise categories are seen during training.

音声認識するニューラルネットの中間の特徴(h)をノイズに対して不変になるように訓練すると、ノイズデータセットが小さくとも精度が上がったという話。音の入力はエンコーダ(E)によりhに変換されるが、このhに音素の識別機Rとノイズの種類の識別機Dがついている。RとDの訓練と並行して、Dが弁別能力をなくするようにEを訓練するというアイデア。


Convolutional Neural Networks (CNNs) are effective models for reducing spectral variations and modeling spectral correlations in acoustic features for automatic speech recognition (ASR). Hybrid speech recognition systems incorporating CNNs with Hidden Markov Models/Gaussian Mixture Models (HMMs/GMMs) have achieved the state-of-the-art in various benchmarks. Meanwhile, Connectionist Temporal Classification (CTC) with Recurrent Neural Networks (RNNs), which is proposed for labeling unsegmented sequences, makes it feasible to train an end-to-end speech recognition system instead of hybrid settings. However, RNNs are computationally expensive and sometimes difficult to train. In this paper, inspired by the advantages of both CNNs and the CTC approach, we propose an end-to-end speech framework for sequence labeling, by combining hierarchical CNNs with CTC directly without recurrent connections. By evaluating the approach on the TIMIT phoneme recognition task, we show that the proposed model is not only computationally efficient, but also competitive with the existing baseline systems. Moreover, we argue that CNNs have the capability to model temporal correlations with appropriate context information.

CNNだけで音声認識を行う試み。LSTMを使ったときより計算的にefficient。TIMITデータセットではLSTMと同程度の精度を出すことができた。


In this paper, we introduce a variation of the skip-gram model which jointly learns distributed word vector representations and their way of composing to form phrase embeddings. In particular, we propose a learning procedure that incorporates a phrase-compositionality function which can capture how we want to compose phrases vectors from their component word vectors. Our experiments show improvement in word and phrase similarity tasks as well as syntactic tasks like dependency parsing using the proposed joint models.

skip-gramを使ってフレーズレベルのベクトルを獲得する話。単語ベクトルと単語ベクトルを組み合わせて作るフレーズベクトルの作り方を同時に学習している。単語類似度とアナロジーおよび係り受け解析で評価した結果、単なるskip-gramより性能が若干向上した。


人間が持っている構造化された事前知識をナレッジグラフの形でディープラーニングのclassificationに導入する手法の提案。Graph Search Neural Networkを使用しEnd-to-Endで学習が可能。


Artificial intelligence has seen several breakthroughs in recent years, with games often serving as milestones. A common feature of these games is that players have perfect information. Poker is the quintessential game of imperfect information, and a longstanding challenge problem in artificial intelligence. We introduce DeepStack, an algorithm for imperfect information settings. It combines recursive reasoning to handle information asymmetry, decomposition to focus computation on the relevant decision, and a form of intuition that is automatically learned from self-play using deep learning. In a study involving 44,000 hands of poker, DeepStack defeated with statistical significance professional poker players in heads-up no-limit Texas hold'em. The approach is theoretically sound and is shown to produce more difficult to exploit strategies than prior approaches.

DNNでポーカーを行い、プロより強くなったという話。ポーカーが、相手の手札がわからない不完全情報ゲームという点でこの意義は大きい。 判断の後悔を最小化するというCFRの考えがベースになっている。ただ、これは当然結末に至る手札がわからないと後悔の程度がわからない。そこで、その推計に相手のあり得る手札を入力としたDNNを使用している。


In the last few years, deep learning has led to very good performance on a variety of problems, such as visual recognition, speech recognition and natural language processing. Among different types of deep neural networks, convolutional neural networks have been most extensively studied. Due to the lack of training data and computing power in early days, it is hard to train a large high-capacity convolutional neural network without overfitting. After the rapid growth in the amount of the annotated data and the recent improvements in the strengths of graphics processor units (GPUs), the research on convolutional neural networks has been emerged swiftly and achieved state-of-the-art results on various tasks. In this paper, we provide a broad survey of the recent advances in convolutional neural networks. Besides, we also introduce some applications of convolutional neural networks in computer vision.

2017年年初に送る、これまでのCNNのまとめ。構成方法、最適化手法から適用先まで、幅広くまとめられている。この図だけでもかなりの価値がある。


Current generative frameworks use end-to-end learning and generate images by sampling from uniform noise distribution. However, these approaches ignore the most basic principle of image formation: images are product of: (a) Structure: the underlying 3D model; (b) Style: the texture mapped onto structure. In this paper, we factorize the image generation process and propose Style and Structure Generative Adversarial Network (S^2-GAN). Our S^2-GAN has two components: the Structure-GAN generates a surface normal map; the Style-GAN takes the surface normal map as input and generates the 2D image. Apart from a real vs. generated loss function, we use an additional loss with computed surface normals from generated images. The two GANs are first trained independently, and then merged together via joint learning. We show our S^2-GAN model is interpretable, generates more realistic images and can be used to learn unsupervised RGBD representations.

GANで生成を行う際に物体の構造が考慮されるべき、とし、表面のテクスチャ(surface normal map)を生成してからそれを入力に画像生成を行う、という二段構えのGANを考案。その名もS2-GAN。これで通常より高い識別性を持つ画像を生成できた。


Despite the loss of semantic information, bag-of-ngram based methods still achieve state-of-the-art results for tasks such as sentiment classification of long movie reviews. Many document embeddings methods have been proposed to capture semantics, but they still can't outperform bag-of-ngram based methods on this task. In this paper, we modify the architecture of the recently proposed Paragraph Vector, allowing it to learn document vectors by predicting not only words, but n-gram features as well. Our model is able to capture both semantics and word order in documents while keeping the expressive power of learned vectors. Experimental results on IMDB movie review dataset shows that our model outperforms previous deep learning models and bag-of-ngram based models due to the above advantages. More robust results are also obtained when our model is combined with other models. The source code of our model will be also published together with this paper.

評価分析を長い映画レビューに対して行うと、既存の文書embeddingではbag-of-ngramを下回るので、超えられる文書embedding手法を提案した話。具体的にはMikolovらが提案したParagraph Vectorの考え方をベースに、文書から単語とbag-of-ngramを予測することで文書ベクトルを獲得している。IMDBのデータセットで評価をしたところ、accuracyで既存の文書embedding手法とbag-of-ngramを上回った。


Recently, a new document metric called the word mover’s distance (WMD) has been proposed with unprecedented results on kNN-based document classification. The WMD elevates high-quality word embeddings to a document metric by formulating the distance between two documents as an optimal transport problem between the embedded words. However, the document distances are entirely unsupervised and lack a mechanism to incorporate supervision when available. In this paper we propose an efficient technique to learn a supervised metric, which we call the Supervised-WMD (S-WMD) metric. The supervised training minimizes the stochastic leave-one-out nearest neighbor classification error on a perdocument level by updating an affine transformation of the underlying word embedding space and a word-imporance weight vector. As the gradient of the original WMD distance would result in an inefficient nested optimization problem, we provide an arbitrarily close approximation that results in a practical and efficient update rule. We evaluate S-WMD on eight real-world text classification tasks on which it consistently outperforms almost all of our 26 competitive baselines.

分類タスクを対象に教師なしの文書間距離指標WMD(#147)の教師あり拡張Supervised-WMD(SWMD)の提案。WMDとの違いは文書に含まれる単語ヒストグラムへの単語ごと重要度の重み付けと埋め込み表現の線形変換を行うことで、この重みと変換行列を訓練用データから学習する。8つのデータセットの文書分類タスクを教師なし、教師あり、WMDを含む26の手法で比較し最もよい性能を示した。


Neural word representations have proven useful in Natural Language Processing (NLP) tasks due to their ability to efficiently model complex semantic and syntactic word relationships. However, most techniques model only one representation per word, despite the fact that a single word can have multiple meanings or "senses". Some techniques model words by using multiple vectors that are clustered based on context. However, recent neural approaches rarely focus on the application to a consuming NLP algorithm. Furthermore, the training process of recent word-sense models is expensive relative to single-sense embedding processes. This paper presents a novel approach which addresses these concerns by modeling multiple embeddings for each word based on supervised disambiguation, which provides a fast and accurate way for a consuming NLP model to select a sense-disambiguated embedding. We demonstrate that these embeddings can disambiguate both contrastive senses such as nominal and verbal senses as well as nuanced senses such as sarcasm. We further evaluate Part-of-Speech disambiguated embeddings on neural dependency parsing, yielding a greater than 8% average error reduction in unlabeled attachment scores across 6 languages.

単語には複数の語義があるのに各単語につき一つの単語ベクトルしか学習していない問題に対して、語義ごとにベクトルを作ることを提案している。手法としてはPoSタガーを用いてタグ付けしたコーパスを用いて、CBOWやSkip-gramで語義を予測させることで学習する。"主観的"な評価の結果、確かに語義ごとのベクトルを得られていた。


We present the Word Mover’s Distance (WMD), a novel distance function between text documents. Our work is based on recent results in word embeddings that learn semantically meaningful representations for words from local cooccurrences in sentences. The WMD distance measures the dissimilarity between two text documents as the minimum amount of distance that the embedded words of one document need to “travel” to reach the embedded words of another document. We show that this distance metric can be cast as an instance of the Earth Mover’s Distance, a well studied transportation problem for which several highly efficient solvers have been developed. Our metric has no hyperparameters and is straight-forward to implement. Further, we demonstrate on eight real world document classification data sets, in comparison with seven stateof-the-art baselines, that the WMD metric leads to unprecedented low k-nearest neighbor document classification error rates.

文書間の距離指標Word Mover's Distance(WMD)の提案。分布間の距離指標Earth Mover's Distance(EMD)をベースに文書に含まれる単語の埋め込みベクトルの集合を分布とみなして距離を算出。k-nnによる文書分類タスクで評価しSOTA手法より低エラーを実現。


Paragraph Vectors has been recently proposed as an unsupervised method for learning distributed representations for pieces of texts. In their work, the authors showed that the method can learn an embedding of movie review texts which can be leveraged for sentiment analysis. That proof of concept, while encouraging, was rather narrow. Here we consider tasks other than sentiment analysis, provide a more thorough comparison of Paragraph Vectors to other document modelling algorithms such as Latent Dirichlet Allocation, and evaluate performance of the method as we vary the dimensionality of the learned representation. We benchmarked the models on two document similarity data sets, one from Wikipedia, one from arXiv. We observe that the Paragraph Vector method performs significantly better than other methods, and propose a simple improvement to enhance embedding quality. Somewhat surprisingly, we also show that much like word embeddings, vector operations on Paragraph Vectors can perform useful semantic results.

Paragraph Vectorの有効性を文書類似度タスクでLDAと比較した話。WikipediaとarXivを対象に比較した結果、LDAと同等かそれ以上の結果を示した。また、Paragraph Vectorでもベクトルの足し引きで意味のある結果(e.g. "Lady Gaga" - "American" + "Japanese" = "Ayumi Hamasaki")を得ることができた。


We introduce a new language learning setting relevant to building adaptive natural language interfaces. It is inspired by Wittgenstein’s language games: a human wishes to accomplish some task (e.g., achieving a certain configuration of blocks), but can only communicate with a computer, who performs the actual actions (e.g., removing all red blocks). The computer initially knows nothing about language and therefore must learn it from scratch through interaction, while the human adapts to the computer’s capabilities. We created a game called SHRDLURN in a blocks world and collected interactions from 100 people playing it. First, we analyze the humans’ strategies, showing that using compositionality and avoiding synonyms correlates positively with task performance. Second, we compare computer strategies, showing that modeling pragmatics on a semantic parsing model accelerates learning for more strategic players.

言語がわからない状態で、人の指示を学習させる試み。ブロックを積み替えるゲームを通じて、言語による指示(赤を消す)->論理構造(remove(red))へのパースを学習させる。


We trained a convolutional neural network (CNN) to map raw pixels from a single front-facing camera directly to steering commands. This end-to-end approach proved surprisingly powerful. With minimum training data from humans the system learns to drive in traffic on local roads with or without lane markings and on highways. It also operates in areas with unclear visual guidance such as in parking lots and on unpaved roads. The system automatically learns internal representations of the necessary processing steps such as detecting useful road features with only the human steering angle as the training signal. We never explicitly trained it to detect, for example, the outline of roads. Compared to explicit decomposition of the problem, such as lane marking detection, path planning, and control, our end-to-end system optimizes all processing steps simultaneously. We argue that this will eventually lead to better performance and smaller systems. Better performance will result because the internal components self-optimize to maximize overall system performance, instead of optimizing human-selected intermediate criteria, e.g., lane detection. Such criteria understandably are selected for ease of human interpretation which doesn't automatically guarantee maximum system performance. Smaller networks are possible because the system learns to solve the problem with the minimal number of processing steps. We used an NVIDIA DevBox and Torch 7 for training and an NVIDIA DRIVE(TM) PX self-driving car computer also running Torch 7 for determining where to drive. The system operates at 30 frames per second (FPS).

NVIDIAによる自動運転。この論文では画像入力からのend-to-endでの走路追従を対象とし出力としてステアリング舵角の値を教師有りの学習をCNNで行う。人間のドライバーの走行データから学習。汎化のため入力画像をaugumentationしそれに対応してステアリング舵角の値も補正し学習する。


Objects appear to scale differently in natural images. This fact requires methods dealing with object-centric tasks (e.g. object proposal) to have robust performance over variances in object scales. In the paper, we present a novel segment proposal framework, namely FastMask, which takes advantage of hierarchical features in deep convolutional neural networks to segment multi-scale objects in one shot. Innovatively, we adapt segment proposal network into three different functional components (body, neck and head). We further propose a weight-shared residual neck module as well as a scale-tolerant attentional head module for efficient one-shot inference. On MS COCO benchmark, the proposed FastMask outperforms all state-of-the-art segment proposal methods in average recall being 2~5 times faster. Moreover, with a slight trade-off in accuracy, FastMask can segment objects in near real time (~13 fps) with 800*600 resolution images, demonstrating its potential in practical applications. Our implementation is available on this https URL.

one-shotで複数のscaleの物体に対応出来るInstanceを考慮したSemantic Segmentation手法であるFastMaskの提案。役割の違うbody, neck, headという3つの構造を組み合わせてネットワークを構築(Fig.2参照)。ネットワークのコアは重み共有したResNeckモジュール。ResNeck構造でfeature mapをズームアウトして効率的に異なるスケールの物体に対応。既存手法と比較して処理スピードは2〜5倍。提案したhead構造(Attention Head)によって、受容野サイズが物体に合っていない事に起因した背景のノイズの影響を減少させることが出来る。SoTA。


Currently, the most successful learning models in computer vision are based on learning successive representations followed by a decision layer. This is usually actualized through feedforward multilayer neural networks, e.g. ConvNets, where each layer forms one of such successive representations. However, an alternative that can achieve the same goal is a feedback based approach in which the representation is formed in an iterative manner based on a feedback received from previous iteration's output. We establish that a feedback based approach has several fundamental advantages over feedforward: it enables making early predictions at the query time, its output naturally conforms to a hierarchical structure in the label space (e.g. a taxonomy), and it provides a new basis for Curriculum Learning. We observe that feedback networks develop a considerably different representation compared to feedforward counterparts, in line with the aforementioned advantages. We put forth a general feedback based learning architecture with the endpoint results on par or better than existing feedforward networks with the addition of the above advantages. We also investigate several mechanisms in feedback architectures (e.g. skip connections in time) and design choices (e.g. feedback length). We hope this study offers new perspectives in quest for more natural and practical learning models.

feedforwardに取って代わる可能性のあるfeedbackネットワーク提案。同一入力に対して繰り返し予測を行い、前の処理の結果を次の処理に反映することでfeedbackネットワークを構築。フィードフォワードと比較した利点として段階的な予測(Early Predictions)が可能である。ラベル空間の階層構造（分類法など）に準拠(Taxnomoy Compliance)しcurriculum learning（簡単なものから順次学んでいく学習）を行う。ネットワークのコアとなる構造はstacking ConvLSTM。lossは各Tでの出力におけるlossと最終出力のlossの荷重平均。CIFAR100を使った実験結果でResNet24超え。


Vector space word representations are learned from distributional information of words in large corpora. Although such statistics are semantically informative, they disregard the valuable information that is contained in semantic lexicons such as WordNet, FrameNet, and the Paraphrase Database. This paper proposes a method for refining vector space representations using relational information from semantic lexicons by encouraging linked words to have similar vector representations, and it makes no assumptions about how the input vectors were constructed. Evaluated on a battery of standard lexical semantic evaluation tasks in several languages, we obtain substantial improvements starting with a variety of word vector models. Our refinement method outperforms prior techniques for incorporating semantic lexicons into the word vector training algorithms.

Skip-gramやGloVeで得た単語ベクトルに対し、WordNetなどの外部知識を用いることで単語ベクトルを洗練する手法を提案。外部知識上で関連する単語を似たベクトルにするために似せたいベクトル間のユークリッド距離を最小化する。意味評価をした結果、一部タスクを除いて性能が向上した。


Synthesizing high resolution photorealistic images has been a long-standing challenge in machine learning. In this paper we introduce new methods for the improved training of generative adversarial networks (GANs) for image synthesis. We construct a variant of GANs employing label conditioning that results in 128x128 resolution image samples exhibiting global coherence. We expand on previous work for image quality assessment to provide two new analyses for assessing the discriminability and diversity of samples from class-conditional image synthesis models. These analyses demonstrate that high resolution samples provide class information not present in low resolution samples. Across 1000 ImageNet classes, 128x128 samples are more than twice as discriminable as artificially resized 32x32 samples. In addition, 84.7% of the classes have samples exhibiting diversity comparable to real ImageNet data.

画像生成について、単に生成するだけでなく、クラスの識別をさせる補助的なタスクをさせることで、識別性能(=解像度)を上げることができたという話。 研究内容はもちろんだが、画像生成について「識別性(=解像度)」と「多様性」という評価軸が設定されていて、この点は今後意識が必要と感じた


While neural networks have been successfully applied to many natural language processing tasks, they come at the cost of interpretability. In this paper, we propose a general methodology to analyze and interpret decisions from a neural model by observing the effects on the model of erasing various parts of the representation, such as input word-vector dimensions, intermediate hidden units, or input words. We present several approaches to analyzing the effects of such erasure, from computing the relative difference in evaluation metrics, to using reinforcement learning to erase the minimum set of input words in order to flip a neural model's decision. In a comprehensive analysis of multiple NLP tasks, including linguistic feature classification, sentence-level sentiment analysis, and document level sentiment aspect prediction, we show that the proposed methodology not only offers clear explanations about neural model decisions, but also provides a way to conduct error analysis on neural models.

自然言語において、ニューラルネットが何をどう判断しているのか解釈する試み。基本はある次元を抜いたときどれだけ尤度に影響があるかを調べることで、隠れ層の重要度ヒートマップを作る。ただ、自然言語においては単語の複合も重要。そこで強化学習を用い、主要要素以外を抜く試みも行っている。


We analyze skip-gram with negative-sampling (SGNS), a word embedding method introduced by Mikolov et al., and show that it is implicitly factorizing a word-context matrix, whose cells are the pointwise mutual information (PMI) of the respective word and context pairs, shifted by a global constant. We find that another embedding method, NCE, is implicitly factorizing a similar matrix, where each cell is the (shifted) log conditional probability of a word given its context. We show that using a sparse Shifted Positive PMI word-context matrix to represent words improves results on two word similarity tasks and one of two analogy tasks. When dense low-dimensional vectors are preferred, exact factorization with SVD can achieve solutions that are at least as good as SGNS’s solutions for word similarity tasks. On analogy questions SGNS remains superior to SVD. We conjecture that this stems from the weighted nature of SGNS’s factorization.

Skip-gram with negative samplingで学習したword embeddingが、ある仮定の下ではPMIの行列を分解しているのと等価なことを示した論文。SPPMIを用いて単語を表現したところ単語類似度タスクとアナロジータスクのうちの一つで性能が向上することを示した。


Context-predicting models (more commonly known as embeddings or neural language models) are the new kids on the distributional semantics block. Despite the buzz surrounding these models, the literature is still lacking a systematic comparison of the predictive models with classic, count-vector-based distributional semantic approaches. In this paper, we perform such an extensive evaluation, on a wide range of lexical semantics tasks and across many parameter settings. The results, to our own surprise, show that the buzz is fully justified, as the context-predicting models obtain a thorough and resounding victory against their count-based counterparts.

単語ベクトルを獲得するための2つの枠組みであるcount-baseとpredict-baseな手法を初めてシステマティックに比較した論文。ハイパーパラメータを変え、analogyやsemantic relatedness含む5つの意味タスクで比較した結果、predict-baseな手法の方が優れているという結果になった。Don't count, predict!


We present weight normalization: a reparameterization of the weight vectors in a neural network that decouples the length of those weight vectors from their direction. By reparameterizing the weights in this way we improve the conditioning of the optimization problem and we speed up convergence of stochastic gradient descent. Our reparameterization is inspired by batch normalization but does not introduce any dependencies between the examples in a minibatch. This means that our method can also be applied successfully to recurrent models such as LSTMs and to noise-sensitive applications such as deep reinforcement learning or generative models, for which batch normalization is less well suited. Although our method is much simpler, it still provides much of the speed-up of full batch normalization. In addition, the computational overhead of our method is lower, permitting more optimization steps to be taken in the same amount of time. We demonstrate the usefulness of our method on applications in supervised image recognition, generative modelling, and deep reinforcement learning.

DNNの重み更新時に重みベクトルをノルムと正規化したベクトルの向きにパラメトライズしそれぞれの勾配から学習するweight normalization(WN)を提案。画像識別、生成モデル、強化学習タスクでbatch normalization(BN)などと比較評価しタスクによらず学習が高速に収束することを示した。


Recent trends suggest that neural-network-inspired word embedding models outperform traditional count-based distributional models on word similarity and analogy detection tasks. We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves. Furthermore, we show that these modifications can be transferred to traditional distributional models, yielding similar gains. In contrast to prior reports, we observe mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others.

word embeddingの性能向上はアルゴリズムでなくハイパーパラメータのチューニングによるところが大きいのではないかということで検証した論文。様々なハイパーパラメータの組み合わせで検証した結果、word similarityタスクにおいて、従来のcount-baseの手法でもpredict-baseの手法と同等の性能を示した。ただし、analogy-taskにおいてはpredict-base手法の方が強かった。


This paper deals with price optimization, which is to find the best pricing strategy that maximizes revenue or profit, on the basis of demand forecasting models. Though recent advances in regression technologies have made it possible to reveal price-demand relationship of a number of multiple products, most existing price optimization methods, such as mixed integer programming formulation, cannot handle tens or hundreds of products because of their high computational costs. To cope with this problem, this paper proposes a novel approach based on network flow algorithms. We reveal a connection between supermodularity of the revenue and cross elasticity of demand. On the basis of this connection, we propose an efficient algorithm that employs network flow algorithms. The proposed algorithm can handle hundreds or thousands of products, and returns an exact optimal solution under an assumption regarding cross elasticity of demand. Even in case in which the assumption does not hold, the proposed algorithm can efficiently find approximate solutions as good as can other state-of-the-art methods, as empirical results show.

商品の価格最適化に関する論文。利益を最大化するには大量の商品に対して価格-需要の関係性を明らかにする必要があるが、混合整数計画問法は計算量的に難しかった。当該論文ではnetwork flow algorithmsを使用することによって計算量の問題を解決。また、収益のsupermodularityと需要の交差弾力性との関係を明らかにし、その関係性による仮定を置くことで計算量を削減。仮説が成り立たない場合でも既存手法と同等の性能。最終的にはBQP(binary quadratic programming)に落とせるのでBQPを解く。


Vector-space word representations have been very successful in recent years at improving performance across a variety of NLP tasks. However, common to most existing work, words are regarded as independent entities without any explicit relationship among morphologically related words being modeled. As a result, rare and complex words are often poorly estimated, and all unknown words are represented in a rather crude way using only one or a few vectors. This paper addresses this shortcoming by proposing a novel model that is capable of building representations for morphologically complex words from their morphemes. We combine recursive neural networks (RNNs), where each morpheme is a basic unit, with neural language models (NLMs) to consider contextual information in learning morphologicallyaware word representations. Our learned models outperform existing word representations by a good margin on word similarity tasks across many datasets, including a new dataset we introduce focused on rare words to complement existing ones in an interesting way.

Recursive Neural Network(RNN)とニューラル言語モデルを組み合わせて形態素から単語ベクトル表現を構築することで、まれ語や複合語、未知語を上手く表現する手法を提案している。単語の類似度タスクで評価した結果、ほとんどすべてのデータセットで従来手法を超える性能を得られた。


We analyze three critical components of word embedding training: the model, the corpus, and the training parameters. We systematize existing neural-network-based word embedding algorithms and compare them using the same corpus. We evaluate each word embedding in three ways: analyzing its semantic properties, using it as a feature for supervised tasks and using it to initialize neural networks. We also provide several simple guidelines for training word embeddings. First, we discover that corpus domain is more important than corpus size. We recommend choosing a corpus in a suitable domain for the desired task, after that, using a larger corpus yields better results. Second, we find that faster models provide sufficient performance in most cases, and more complex models can be used if the training corpus is sufficiently large. Third, the early stopping metric for iterating should rely on the development set of the desired task rather than the validation loss of training embedding.

word embeddingの学習に重要なコンポーネントをモデル、コーパス、パラメータの3つとし、既存の手法を分類・比較した論文。比較評価を行い、良いword embeddingを学習するためのガイドラインを示した。


The pre-dominant approach to language modeling to date is based on recurrent neural networks. Their success on this task is often linked to their ability to capture unbounded context. In this paper we develop a finite context approach through stacked convolutions, which can be more efficient since they allow parallelization over sequential tokens. We propose a novel simplified gating mechanism that outperforms Oord et al (2016) and investigate the impact of key architectural decisions. The proposed approach achieves state-of-the-art on the WikiText-103 benchmark, even though it features long-term dependencies, as well as competitive results on the Google Billion Words benchmark. Our model reduces the latency to score a sentence by an order of magnitude compared to a recurrent baseline. To our knowledge, this is the first time a non-recurrent approach is competitive with strong recurrent models on these large scale language tasks.

言語モデルのタスクで、CNNでLSTM同等以上の精度を出したという話。畳み込んだ結果をGRUに近い機構で処理し、過去の情報が消失しないようにしている。Google Billion Wordのデータセットでは、LSTMと同等の精度を出す一方計算効率が20倍程度改善された。


With recent progress in graphics, it has become more tractable to train models on synthetic images, potentially avoiding the need for expensive annotations. However, learning from synthetic images may not achieve the desired performance due to a gap between synthetic and real image distributions. To reduce this gap, we propose Simulated+Unsupervised (S+U) learning, where the task is to learn a model to improve the realism of a simulator's output using unlabeled real data, while preserving the annotation information from the simulator. We develop a method for S+U learning that uses an adversarial network similar to Generative Adversarial Networks (GANs), but with synthetic images as inputs instead of random vectors. We make several key modifications to the standard GAN algorithm to preserve annotations, avoid artifacts, and stabilize training: (i) a 'self-regularization' term, (ii) a local adversarial loss, and (iii) updating the discriminator using a history of refined images. We show that this enables generation of highly realistic images, which we demonstrate both qualitatively and with a user study. We quantitatively evaluate the generated images by training models for gaze estimation and hand pose estimation. We show a significant improvement over using synthetic images, and achieve state-of-the-art results on the MPIIGaze dataset without any labeled real data.

合成画像で学習できると良いけど独特のクセにより上手くいかない、という点を克服する試み。より「本物らしく」するNN対見破るNNで学習(GAN)。元からかけ離れた「本物化」を防ぐため元画像との差異を利用した正規化などの試みがとられている


Reasoning about objects, relations, and physics is central to human intelligence, and a key goal of artificial intelligence. Here we introduce the interaction network, a model which can reason about how objects in complex systems interact, supporting dynamical predictions, as well as inferences about the abstract properties of the system. Our model takes graphs as input, performs object- and relation-centric reasoning in a way that is analogous to a simulation, and is implemented using deep neural networks. We evaluate its ability to reason about several challenging physical domains: n-body problems, rigid-body collision, and non-rigid dynamics. Our results show it can be trained to accurately simulate the physical trajectories of dozens of objects over thousands of time steps, estimate abstract quantities such as energy, and generalize automatically to systems with different numbers and configurations of objects and relations. Our interaction network implementation is the first general-purpose, learnable physics engine, and a powerful general framework for reasoning about object and relations in a wide variety of complex real-world domains.

多体問題などの相互作用のある複雑な物理法則を物体とその関係性に分解することで、推論可能とした手法の提案。ネットワークへの入力、は物体（ノード）とノード間の関係（エッジ）のグラフ。1つ目の近似関数（ネットワーク）で物体に対する効果量を計算し、外力とともに相互的な影響を加味した状態推定を行う関数（ネットワークに）に投入される。計算量的に大規模なものには対応出来ていないが、手法としては汎用性が高く一般的な多くの物理現象に適用可能。


Several machine learning tasks require to represent the data using only a sparse set of interest points. An ideal detector is able to find the corresponding interest points even if the data undergo a transformation typical for a given domain. Since the task is of high practical interest in computer vision, many hand-crafted solutions were proposed. In this paper, we ask a fundamental question: can we learn such detectors from scratch? Since it is often unclear what points are "interesting", human labelling cannot be used to find a truly unbiased solution. Therefore, the task requires an unsupervised formulation. We are the first to propose such a formulation: training a neural network to rank points in a transformation-invariant manner. Interest points are then extracted from the top/bottom quantiles of this ranking. We validate our approach on two tasks: standard RGB image interest point detection and challenging cross-modal interest point detection between RGB and depth images. We quantitatively show that our unsupervised method performs better or on-par with baselines.

位置, 回転, スケールについて不変量と共変量を考慮したDLによる特徴点抽出手法を提案。性能としては既存手法と同程度だが、クロスモーダルなデータ間で再現可能性があることが特徴。学習方法は、2つの異なるアングルの画像で同一位置のpatchを切り出して、forward処理。response functionを算出し、2つの値をhingi lossで評価。完全な教師なし学習では、同一画像で異なるaugmentationを掛けたものを入力として評価している。baselineはDoG。


The cost of large scale data collection and annotation often makes the application of machine learning algorithms to new tasks or datasets prohibitively expensive. One approach circumventing this cost is training models on synthetic data where annotations are provided automatically. Despite their appeal, such models often fail to generalize from synthetic to real images, necessitating domain adaptation algorithms to manipulate these models before they can be successfully applied. Existing approaches focus either on mapping representations from one domain to the other, or on learning to extract features that are invariant to the domain from which they were extracted. However, by focusing only on creating a mapping or shared representation between the two domains, they ignore the individual characteristics of each domain. We suggest that explicitly modeling what is unique to each domain can improve a model's ability to extract domain-invariant features. Inspired by work on private-shared component analysis, we explicitly learn to extract image representations that are partitioned into two subspaces: one component which is private to each domain and one which is shared across domains. Our model is trained not only to perform the task we care about in the source domain, but also to use the partitioned representation to reconstruct the images from both domains. Our novel architecture results in a model that outperforms the state-of-the-art on a range of unsupervised domain adaptation scenarios and additionally produces visualizations of the private and shared representations enabling interpretation of the domain adaptation process.

DNNのラベル無しデータをターゲットとしたドメインアダプテーションにおいてネットワークをドメイン依存部分とドメイン共通部分を分離するようにソース、ターゲット両データで学習し適用するネットワーク構造の提案。識別はドメイン共通素性ネットワークを使う。教師なしドメインアダプテーションのSoTA。


Recent work has shown that neuralembedded word representations capture many relational similarities, which can be recovered by means of vector arithmetic in the embedded space. We show that Mikolov et al.’s method of first adding and subtracting word vectors, and then searching for a word similar to the result, is equivalent to searching for a word that maximizes a linear combination of three pairwise word similarities. Based on this observation, we suggest an improved method of recovering relational similarities, improving the state-of-the-art results on two recent word-analogy datasets. Moreover, we demonstrate that analogy recovery is not restricted to neural word embeddings, and that a similar amount of relational similarities can be recovered from traditional distributional word representations.

word embeddingにおけるアナロジータスクを解くための類似度計算手法を検証した論文。検証の結果、Neural embeddingでなく従来の単語共起を用いた手法でも、類似度計算手法によってはstate-of-the-artな結果を出せることがわかった。


Efficient simulation of the Navier-Stokes equations for fluid flow is a long standing problem in applied mathematics, for which state-of-the-art methods require large compute resources. In this work, we propose a data-driven approach that leverages the approximation power of deep-learning with the precision of standard solvers to obtain fast and highly realistic simulations. Our method solves the incompressible Euler equations using the standard operator splitting method, in which a large sparse linear system with many free parameters must be solved. We use a Convolutional Network with a highly tailored architecture, trained using a novel unsupervised learning framework to solve the linear system. We present real-time 2D and 3D simulations that outperform recently proposed data-driven methods; the obtained results are realistic and show good generalization properties.

オイラー的手法（差分法）とCNNを組み合わせて、非圧縮流体シミュレーションを行う事を提案。CNNを使っているのは通常線形凸最適化手法が適用される圧力算出部分。CNNへの入力は現在の格子点の速度, 位置, 圧力。NS方程式の粘性項を無視。半教師あり学習で使われている手法をloss(推定速度のダイバージェンス)に導入。これにより収束性が相当改善（入れないと収束しない）。計算速度的にはSoTA。（煙のような）スパースな流体で威力を発揮する。


Traditional fluid simulations require large computational resources even for an average sized scene with the main bottleneck being a very small time step size, required to guarantee the stability of the solution. Despite a large progress in parallel computing and efficient algorithms for pressure computation in the recent years, realtime fluid simulations have been possible only under very restricted conditions. In this paper we propose a novel machine learning based approach, that formulates physics-based fluid simulation as a regression problem, estimating the acceleration of every particle for each frame. We designed a feature vector, directly modelling individual forces and constraints from the Navier-Stokes equations, giving the method strong generalization properties to reliably predict positions and velocities of particles in a large time step setting on yet unseen test videos. We used a regression forest to approximate the behaviour of particles observed in the large training set of simulations obtained using a traditional solver. Our GPU implementation led to a speed-up of one to three orders of magnitude compared to the state-of-the-art position-based fluid solver and runs in real-time for systems with up to 2 million particles.

粒子法による流体シミュレーションをRegression Forestを使った回帰モデルで近似的に行い高速化した研究。入力は各粒子の位置と速度。NS方程式を基に作られたモデルによって、各粒子の位置と速度から特徴量を生成。特徴量は圧力、表面張力、粘性、非圧縮性制約に関連したもの（圧力などそれ自体の値を求めてはいない）。これをRegression Forestに入力し各粒子の加速度（または速度）の推定を行い、時間ステップを使って数値積分をして、次ステップの粒子位置と速度を計算。速度的には従来手法(PBF)の200倍強高速。


Semantic parsing aims at mapping natural language to machine interpretable meaning representations. Traditional approaches rely on high-quality lexicons, manually-built templates, and linguistic features which are either domain- or representation-specific. In this paper we present a general method based on an attention-enhanced encoder-decoder model. We encode input utterances into vector representations, and generate their logical forms by conditioning the output sequences or trees on the encoding vectors. Experimental results on four datasets show that our approach performs competitively without using hand-engineered features and is easy to adapt across domains and meaning representations.

Encoder-Decoderモデル(with Attention)を使って、文を論理式に変換するという研究(例: 面先が一番大きい県の人口は？⇒(人口: i (argmax _ (県:k ) (面積:i )))) など。Lisp的。


Collecting well-annotated image datasets to train modern machine learning algorithms is prohibitively expensive for many tasks. One appealing alternative is rendering synthetic data where ground-truth annotations are generated automatically. Unfortunately, models trained purely on rendered images often fail to generalize to real images. To address this shortcoming, prior work introduced unsupervised domain adaptation algorithms that attempt to map representations between the two domains or learn to extract features that are domain-invariant. In this work, we present a new approach that learns, in an unsupervised manner, a transformation in the pixel space from one domain to the other. Our generative adversarial network (GAN)-based method adapts source-domain images to appear as if drawn from the target domain. Our approach not only produces plausible samples, but also outperforms the state-of-the-art on a number of unsupervised domain adaptation scenarios by large margins. Finally, we demonstrate that the adaptation process generalizes to object classes unseen during training.

ラベル付けされたデータを作るのはとても大変だから、データのあるソースからデータのあまりないターゲットの画像が作れたらいいよね、という話。ソース⇒ターゲットの画像生成を行うGANを作成し、ドメイン適用を実現。生成画像だけを使った学習で、ほかの手法に対し最高精度を達成。


Recurrent Neural Networks (RNNs) have become the state-of-the-art choice for extracting patterns from temporal sequences. However, current RNN models are ill-suited to process irregularly sampled data triggered by events generated in continuous time by sensors or other neurons. Such data can occur, for example, when the input comes from novel event-driven artificial sensors that generate sparse, asynchronous streams of events or from multiple conventional sensors with different update intervals. In this work, we introduce the Phased LSTM model, which extends the LSTM unit by adding a new time gate. This gate is controlled by a parametrized oscillation with a frequency range that produces updates of the memory cell only during a small percentage of the cycle. Even with the sparse updates imposed by the oscillation, the Phased LSTM network achieves faster convergence than regular LSTMs on tasks which require learning of long sequences. The model naturally integrates inputs from sensors of arbitrary sampling rates, thereby opening new areas of investigation for processing asynchronous sensory events that carry timing information. It also greatly improves the performance of LSTMs in standard RNN applications, and does so with an order-of-magnitude fewer computes at runtime.

何かのイベント(手を上げたらとか)をトリガに発生する時系列データは、挙動があるタイミングが疎のため既存のLSTMでは学習が難しい(混在するならなおさら)。そこで、時間情報から今ONなのかOFFなのかを制御するPhaseゲートを設けて学習する話。TensorFlow実装有


Violence is a serious problems for cities like Chicago and has been exacerbated by the use of social media by gang-involved youths for taunting rival gangs. We present a corpus of tweets from a young and powerful female gang member and her communicators, which we have annotated with discourse intention, using a deep read to understand how and what triggered conversations to escalate into aggression. We use this corpus to develop a part-of-speech tagger and phrase table for the variant of English that is used and a classifier for identifying tweets that express grieving and aggression.

ギャングによる暴力を未然に検出、抑制する研究。ギャングにおける中心的な女性メンバと若手のメンバのTwitterのやり取りを分析。どのような対話がエスカレートしやすいのかについて、アノテーションを行いコーパスを作成した。タグは侵略(aggression=相手の攻撃)、喪失(loss=仲間の死など)といったものが付与されている。


While continuous word embeddings are gaining popularity, current models are based solely on linear contexts. In this work, we generalize the skip-gram model with negative sampling introduced by Mikolov et al. to include arbitrary contexts. In particular, we perform experiments with dependency-based contexts, and show that they produce markedly different embeddings. The dependencybased embeddings are less topical and exhibit more functional similarity than the original skip-gram embeddings.

Skip-gramのコンテキストは周辺語しか考えていないけどそれを一般化する話。具体的には、コンテキストとして依存関係を用いて実験を行った。実験を行ったところ、依存関係ベースのembeddingsは周辺語ベースのものに比べて機能的類似性が高いことがわかった。


We describe a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense (grammatically and semantically) using a language model. The entire network is trained jointly on all these tasks using weight-sharing, an instance of multitask learning. All the tasks use labeled data except the language model which is learnt from unlabeled text and represents a novel form of semi-supervised learning for the shared tasks. We show how both multitask learning and semi-supervised learning improve the generalization of the shared tasks, resulting in stateof-the-art performance.

NLPの様々なタスク(NER, POS, チャンキング, 言語モデル, SRL)を同時に学習させることで汎化性能を向上させようという話。具体的には同時に学習するためのニューラルネットワークの枠組みを提案している。結果として、タスク単体で学習させるより複数のタスクを同時に学習させた方が良い結果となった。


This paper presents our recent work on the design and development of a new, large scale dataset, which we name MS MARCO, for MAchine Reading COmprehension.This new dataset is aimed to overcome a number of well-known weaknesses of previous publicly available datasets for the same task of reading comprehension and question answering. In MS MARCO, all questions are sampled from real anonymized user queries. The context passages, from which answers in the dataset are derived, are extracted from real web documents using the most advanced version of the Bing search engine. The answers to the queries are human generated. Finally, a subset of these queries has multiple answers. We aim to release one million queries and the corresponding answers in the dataset, which, to the best of our knowledge, is the most comprehensive real-world dataset of its kind in both quantity and quality. We are currently releasing 100,000 queries with their corresponding answers to inspire work in reading comprehension and question answering along with gathering feedback from the research community.

Microsoftが公開した質問応答のデータセット(10万件)。質問/回答が、人間のものである点が特徴(Bing=検索エンジンへの入力なのでどこまで質問っぽいかは要確認)。回答はBingの検索結果から抜粋して作成。