The United States spends more than $1B each year on initiatives such as the American Community Survey (ACS), a labor-intensive door-to-door study that measures statistics relating to race, gender, education, occupation, unemployment, and other demographic factors. Although a comprehensive source of data, the lag between demographic changes and their appearance in the ACS can exceed half a decade. As digital imagery becomes ubiquitous and machine vision techniques improve, automated data analysis may provide a cheaper and faster alternative. Here, we present a method that determines socioeconomic trends from 50 million images of street scenes, gathered in 200 American cities by Google Street View cars. Using deep learning-based computer vision techniques, we determined the make, model, and year of all motor vehicles encountered in particular neighborhoods. Data from this census of motor vehicles, which enumerated 22M automobiles in total (8% of all automobiles in the US), was used to accurately estimate income, race, education, and voting patterns, with single-precinct resolution. (The average US precinct contains approximately 1000 people.) The resulting associations are surprisingly simple and powerful. For instance, if the number of sedans encountered during a 15-minute drive through a city is higher than the number of pickup trucks, the city is likely to vote for a Democrat during the next Presidential election (88% chance); otherwise, it is likely to vote Republican (82%). Our results suggest that automated systems for monitoring demographic trends may effectively complement labor-intensive approaches, with the potential to detect trends with fine spatial resolution, in close to real time.

都市の統計調査(人種や性別、職業や失業率など・・・)は、個別訪問ベースで行われるため調査結果が出るまで半年ぐらいかかることもあり、実態と乖離することも多い。そこで、街の様子、具体的には街中を走る車の車種からこれらを推計しようという試み。2200万ほどの車種をモデルに利用することで、驚くほど精確に推定ができたという話。 例：15分の中で、セダンの数がピックアップトラックの数よりも多い場合、市は次の大統領選挙で民主党に投票する可能性が高い(88％の確率)。その逆は共和党に投票する可能性が高い(82％)など。


Two recently developed methods, Feedback Alignment (FA) and Direct Feedback Alignment (DFA), have been shown to obtain surprising performance on vision tasks by replacing the traditional backpropagation update with a random feedback update. However, it is still not clear what mechanisms allow learning to happen with these random updates.      In this work we argue that DFA can be viewed as a noisy variant of a layer-wise training method we call Linear Aligned Feedback Systems (LAFS). We support this connection theoretically by comparing the update rules for the two methods.  We additionally empirically verify that the random update matrices used in DFA work effectively as readout matrices, and that strong correlations exist between the error vectors used in the DFA and LAFS updates. With this new connection between DFA and LAFS we are able to explain why the "alignment" happens in DFA.

Back propagationは生体的な学習のプロセスとは乖離があるので、もうちょい実際の生物寄りな学習プロセスを実装するとどうか、という話。ここではDFAというBackpropではなく直接重みを更新する手法と、レイヤ個別に学習させるLAFSがほぼ等価であるという紹介をしている


Artificial neural networks are remarkably adept at sensory processing, sequence learning and reinforcement learning, but are limited in their ability to represent variables and data structures and to store data over long timescales, owing to the lack of an external memory. Here we introduce a machine learning model called a differentiable neural computer (DNC), which consists of a neural network that can read from and write to an external memory matrix, analogous to the random-access memory in a conventional computer. Like a conventional computer, it can use its memory to represent and manipulate complex data structures, but, like a neural network, it can learn to do so from data. When trained with supervised learning, we demonstrate that a DNC can successfully answer synthetic questions designed to emulate reasoning and inference problems in natural language. We show that it can learn tasks such as finding the shortest path between specified points and inferring the missing links in randomly generated graphs, and then generalize these tasks to specific graphs such as transport networks and family trees. When trained with reinforcement learning, a DNC can complete a moving blocks puzzle in which changing goals are specified by sequences of symbols. Taken together, our results demonstrate that DNCs have the capacity to solve complex, structured tasks that are inaccessible to neural networks without external read–write memory.

ニューラルネットによるコンピュータの模倣。 微分可能な外部メモリをもつニューラルネットアーキテクチャ"Defferentiable Neural Computer(DNC)"を開発。Controller Networkにメモリの使い方を学習させることで、構造的なデータを長期的に保持、利用できるようになった。質問文への解答、グラフによる推論、強化学習によるパズルの解決などができる。以前同じくDeepMindが発表したNeural Turing Machineに似ているが、メモリの動的な割当や解放が出来るようになった点で進歩している。


Word embeddings are increasingly used in natural language understanding tasks requiring sophisticated semantic information. However, the quality of new embedding methods is usually evaluated based on simple word similarity benchmarks. We propose evaluating word embeddings in vivo by evaluating them on a suite of popular downstream tasks. To ensure the ease of use of the evaluation, we take care to find a good point in the tradeoff space between (1) creating a thorough evaluation – i.e., we evaluate on a diverse set of tasks; and (2) ensuring an easy and fast evaluation – by using simple models with few tuned hyperparameters. This allows us to release this evaluation as a standardized script and online evaluation, available at http://veceval.com/.

単語表現を評価するためには単語類似性とアナロジータスクがよく使われている。しかし、それらで良い結果でも実際のタスクで良い結果になるとは限らないことが先行研究で示されている。そのあたりを考慮してこの論文ではより実際のタスクに近い評価手法を提案している。具体的にはユーザが訓練した単語ベクトルを6つのタスク(NER, POS, Chunking, Sentiment Analysis, Question Classification, NLI)で評価するためのシステムを構築している。これにより、学習した単語表現が実際に行いたいタスクに近いタスクで有効なのかを簡単に検証できる。このシステムはhttp://veceval.com/で利用できる。


We suggest a new method for creating and using gold-standard datasets for word similarity evaluation. Our goal is to improve the reliability of the evaluation, and we do this by redesigning the annotation task to achieve higher inter-rater agreement, and by defining a performance measure which takes the reliability of each annotation decision in the dataset into account.

単語表現を評価するためには単語類似度のデータセットがよく使われている。しかし、既存のデータセットには2つの問題がある。一つは単語の関連性と類似性を区別していない点、もう一つは評価者間でアノテーションスコアがばらつく点である。これらの問題に対処するために単語類似度のためのデータセットを作成・使用する方法を提案している。アノテーションと性能指標を再設計することで、評価の信頼性を改善した。ヘブライ語に対する２つのデータセットを作成したところ、より高い評価者間の一致を達成。モデルに対する細かい分析を行うことができることを示した。


Many real world tasks such as reasoning and physical interaction require identification and manipulation of conceptual entities. A first step towards solving these tasks is the automated discovery of distributed symbol-like representations. In this paper, we explicitly formalize this problem as inference in a spatial mixture model where each component is parametrized by a neural network. Based on the Expectation Maximization framework we then derive a differentiable clustering method that simultaneously learns how to group and represent individual entities. We evaluate our method on the (sequential) perceptual grouping task and find that it is able to accurately recover the constituent objects. We demonstrate that the learned representations are useful for next-step prediction.

Neural Expectation Maximization (N-EM)の提案。EMアルゴリズムにおけるハイパパラメタ推定にニューラルネットワークを使用(RNN-EM)。RNNのhidden stateの更新がM stepに該当。


The method introduced in this paper aims at helping computer vision practitioners faced with an overfit problem. The idea is to replace, in a 3-branch ResNet, the standard summation of residual branches by a stochastic affine combination. The largest tested model improves on the best single shot published result on CIFAR-10 by reaching 2.86% test error. Code is available at https://github.com/xgastaldi/shake-shake

画像認識のモデルで、ResNetの2つのブロックからの出力をランダムに組み合わせる(Shake)モデルを提案。forward/backward共にShakeすることでCIFAR-10で2.72%のエラーレートを記録(しかもバッチ単位より個別の画像単位で適用したほうが精度が高い)。


We present Char2Wav, an end-to-end model for speech synthesis. Char2Wav has two components: a reader and a neural vocoder. The reader is an encoder-decoder model with attention. The encoder is a bidirectional recurrent neural network that accepts text or phonemes as inputs, while the decoder is a recurrent neural network (RNN) with attention that produces vocoder acoustic features. Neural vocoder refers to a conditional extension of SampleRNN which generates raw waveform samples from intermediate representations. Unlike traditional models for speech synthesis, Char2Wav learns to produce audio directly from text.

これまでの音声合成は、テキスト→発音特徴、発音特徴→音声と二段階に分かれていたが(WaveNetは後者に相当)、これを統合しEnd to Endな音声合成モデルを作成するという話。Attentionを積んだRNNでEncode、階層上のRNN(SampleRNN)でデコードする。他手法との比較結果はまだ記述されてないが、公式サイトで合成された音声を聞くことができる。


Mastering a video game requires skill, tactics and strategy. While these attributes may be acquired naturally by human players, teaching them to a computer program is a far more challenging task. In recent years, extensive research was carried out in the field of reinforcement learning and numerous algorithms were introduced, aiming to learn how to perform human tasks such as playing video games. As a result, the Arcade Learning Environment (ALE) (Bellemare et al., 2013) has become a commonly used benchmark environment allowing algorithms to train on various Atari 2600 games. In many games the state-of-the-art algorithms outperform humans. In this paper we introduce a new learning environment, the Retro Learning Environment --- RLE, that can run games on the Super Nintendo Entertainment System (SNES), Sega Genesis and several other gaming consoles. The environment is expandable, allowing for more video games and consoles to be easily added to the environment, while maintaining the same interface as ALE. Moreover, RLE is compatible with Python and Torch. SNES games pose a significant challenge to current algorithms due to their higher level of complexity and versatility.

SNES(=スーファミの実行環境)で強化学習ができるようにする学習環境RLEを作った話。Atariよりもゲーム数が多く、またバリエーションにも非常に富んでいる。


We introduce a new audio processing technique that increases the sampling rate of signals such as speech or music using deep convolutional neural networks. Our model is trained on pairs of low and high-quality audio examples; at test-time, it predicts missing samples within a low-resolution signal in an interpolation process similar to image super-resolution. Our method is simple and does not involve specialized audio processing techniques; in our experiments, it outperforms baselines on standard speech and music benchmarks at upscaling ratios of 2x, 4x, and 6x. The method has practical applications in telephony, compression, and text-to-speech generation; it demonstrates the effectiveness of feed-forward convolutional architectures on an audio generation task.

画像ならぬ音の超解像に挑戦する研究。端的には低いサンプリングレートの音を畳み込み(ResBlock)＋UpSamplingして高サンプリングレートの音に変換する。これで電話の音声などを受信側で高音質にするといったことが可能になる。発音データセットでPSNR40程度でまあ良しの結果


In this paper, we propose a very deep fully convolutional encoding-decoding framework for image restoration such as denoising and super-resolution. The network is composed of multiple layers of convolution and de-convolution operators, learning end-to-end mappings from corrupted images to the original ones. The convolutional layers act as the feature extractor, which capture the abstraction of image contents while eliminating noises/corruptions. De-convolutional layers are then used to recover the image details. We propose to symmetrically link convolutional and de-convolutional layers with skip-layer connections, with which the training converges much faster and attains a higher-quality local optimum. First, The skip connections allow the signal to be back-propagated to bottom layers directly, and thus tackles the problem of gradient vanishing, making training deep networks easier and achieving restoration performance gains consequently. Second, these skip connections pass image details from convolutional layers to de-convolutional layers, which is beneficial in recovering the original image. Significantly, with the large capacity, we can handle different levels of noises using a single model. Experimental results show that our network achieves better performance than all previously reported state-of-the-art methods.

一枚の画像からその解像度を上げる単画像超解像の論文。 ILSVRC 2015のWinnerであるResNetの手法を取り入れて30層という当時最もディープなモデルを提案し最高性能を獲得し。シンプルでdeepなモデルが良いというトレンドを実証した。


We present a highly accurate single-image super-resolution (SR) method. Our method uses a very deep convolutional network inspired by VGG-net used for ImageNet classification \cite{simonyan2015very}. We find increasing our network depth shows a significant improvement in accuracy. Our final model uses 20 weight layers. By cascading small filters many times in a deep network structure, contextual information over large image regions is exploited in an efficient way. With very deep networks, however, convergence speed becomes a critical issue during training. We propose a simple yet effective training procedure. We learn residuals only and use extremely high learning rates (104 times higher than SRCNN \cite{dong2015image}) enabled by adjustable gradient clipping. Our proposed method performs better than existing methods in accuracy and visual improvements in our results are easily noticeable.

一枚の画像からその解像度を上げる単画像超解像の論文。 ILSVRC 2015のWinnerであるResNetの手法を取り入れて残差項の推定に特化し、学習係数を従来の1万倍に設定し非常に早い時点での収束を実現した。それでいて性能もほぼ最高性能であった。


We propose an image super-resolution method (SR) using a deeply-recursive convolutional network (DRCN). Our network has a very deep recursive layer (up to 16 recursions). Increasing recursion depth can improve performance without introducing new parameters for additional convolutions. Albeit advantages, learning a DRCN is very hard with a standard gradient descent method due to exploding/vanishing gradients. To ease the difficulty of training, we propose two extensions: recursive-supervision and skip-connection. Our method outperforms previous methods by a large margin.

一枚の画像からその解像度を上げる単画像超解像の論文。 16枚の同じCNNを繰り返し適用して少しづつ高解像画像の推定を行なっている。プーリングを利用せずに合計で20枚ものCNNを適用しているが、同じCNNパラメータを利用することで学習の発散を抑えているのが特徴。


We propose a deep learning method for single image super-resolution (SR). Our method directly learns an end-to-end mapping between the low/high-resolution images. The mapping is represented as a deep convolutional neural network (CNN) that takes the low-resolution image as the input and outputs the high-resolution one. We further show that traditional sparse-coding-based SR methods can also be viewed as a deep convolutional network. But unlike traditional methods that handle each component separately, our method jointly optimizes all layers. Our deep CNN has a lightweight structure, yet demonstrates state-of-the-art restoration quality, and achieves fast speed for practical on-line usage. We explore different network structures and parameter settings to achieve trade-offs between performance and speed. Moreover, we extend our network to cope with three color channels simultaneously, and show better overall reconstruction quality.

一枚の画像からその解像度を上げる単画像超解像の論文。DLを使った超解像の研究の起点になったものと思われる。3枚から4枚のCNNを用いてfeature mapを作成し高解像度の画像を生成する。


We consider the general problem of modeling temporal data with long-range dependencies, wherein new observations are fully or partially predictable based on temporally-distant, past observations. A sufficiently powerful temporal model should separate predictable elements of the sequence from unpredictable elements, express uncertainty about those unpredictable elements, and rapidly identify novel elements that may help to predict the future. To create such models, we introduce Generative Temporal Models augmented with external memory systems. They are developed within the variational inference framework, which provides both a practical training methodology and methods to gain insight into the models' operation. We show, on a range of problems with sparse, long-term temporal dependencies, that these models store information from early in a sequence, and reuse this stored information efficiently. This allows them to perform substantially better than existing models based on well-known recurrent neural networks, like LSTMs.

時系列データのモデリング方法についての提案。RNNにより潜在変数を予測する生成モデル(Variational RNN)をベースに、各タイムステップにおける潜在変数を格納したメモリ(=時間ごとのデータのふるまいの記憶)を用意し、Attentionによりそこから読み出す機構を提案。


An Empirical Exploration of Recurrent Network Architectures

LSTM/GRUよりも優れた構造を持つものはないか？ということを検証した論文。過去の実績から100の優れた構造をピックアップし、これらとLSTM/GRUを4つのタスク(数式意味解釈、XMLタグ予測、言語モデル、音楽生成)でハイパーパラメーターを変えながら検証。結果、LSTM/GRUをすべてのタスクで上回るモデルは発見できなかった。なお、LSTMのforget gateのbiasは1にするとすごいい。


Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning mid- and long-range dependencies. However, conventional attention mechanisms used in memory-augmented neural language models produce a single output vector per time step. This vector is used both for predicting the next token as well as for the key and value of a differentiable memory of a token history. In this paper, we propose a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution. This model outperforms existing memory-augmented neural language models on two corpora. Yet, we found that our method mainly utilizes a memory of the five most recent output representations. This led to the unexpected main finding that a much simpler model based only on the concatenation of recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models.

Attentionを行う場合、隠れ層のベクトルは次の単語の予測・Attentionの算出・将来の単語に有用な情報の格納、という3つの役割を担っていることになる。なので出力を3つにして役割分担させるアイデア。併せて、単純に過去の隠れ層を結合して入力するだけでも高精度になることを確認


Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.

Batch Normalizationの改良版。バッチサイズが小さい時の問題、また推定時と学習時とで正規化方法に差異が出る問題を解消する試み。具体的には、最初は通常通りバッチ内で正規化するけど、徐々にデータ全体の正規化パラメーター(移動平均/分散)へシフトしていくという手法。


Recently, methods have been proposed that perform texture synthesis and style transfer by using convolutional neural networks (e.g. Gatys et al. [2015,2016]). These methods are exciting because they can in some cases create results with state-of-the-art quality. However, in this paper, we show these methods also have limitations in texture quality, stability, requisite parameter tuning, and lack of user controls. This paper presents a multiscale synthesis pipeline based on convolutional neural networks that ameliorates these issues. We first give a mathematical explanation of the source of instabilities in many previous approaches. We then improve these instabilities by using histogram losses to synthesize textures that better statistically match the exemplar. We also show how to integrate localized style losses in our multiscale framework. These losses can improve the quality of large features, improve the separation of content and style, and offer artistic controls such as paint by numbers. We demonstrate that our approach offers improved quality, convergence in fewer iterations, and more stability over the optimization.

StyleTransferを改善する方法についての論文。表現方法(=色の「分布」)の差異を反映できるよう元画像のヒストグラムとの差異を表すlossの項を導入、絵の中のどのスタイルをターゲットのどこに適用するか指示できるようにする、またパラメーターの自動調整を行うといった3本立て。


Previous studies support the idea of merging auditory-based Gabor features with deep learning architectures to achieve robust automatic speech recognition, however, the cause behind the gain of such combination is still unknown. We believe these representations provide the deep learning decoder with more discriminable cues. Our aim with this paper is to validate this hypothesis by performing experiments with three different recognition tasks (Aurora 4, CHiME 2 and CHiME 3) and assess the discriminability of the information encoded by Gabor filterbank features. Additionally, to identify the contribution of low, medium and high temporal modulation frequencies subsets of the Gabor filterbank were used as features (dubbed LTM, MTM and HTM respectively). With temporal modulation frequencies between 16 and 25 Hz, HTM consistently outperformed the remaining ones in every condition, highlighting the robustness of these representations against channel distortions, low signal-to-noise ratios and acoustically challenging real-life scenarios with relative improvements from 11 to 56% against a Mel-filterbank-DNN baseline. To explain the results, a measure of similarity between phoneme classes from DNN activations is proposed and linked to their acoustic properties. We find this measure to be consistent with the observed error rates and highlight specific differences on phoneme level to pinpoint the benefit of the proposed features.

音声認識では入力特徴量としてMel filterbankを使うより、脳と同様にGaborフィルタを使った方が認識精度が良いという話。WERが11から56%の向上。Aurora4, ChiME2, CHiME3データセット。なおGaborの方が良いという話はこの論文が初めてではなくて、Gaborのどれが良いかをいろいろ比較したということ。


In recent years, machine learning techniques based on neural networks for mobile computing become increasingly popular. Classical multi-layer neural networks require matrix multiplications at each stage. Multiplication operation is not an energy efficient operation and consequently it drains the battery of the mobile device. In this paper, we propose a new energy efficient neural network with the universal approximation property over space of Lebesgue integrable functions. This network, called, additive neural network, is very suitable for mobile computing. The neural structure is based on a novel vector product definition, called ef-operator, that permits a multiplier-free implementation. In ef-operation, the "product" of two real numbers is defined as the sum of their absolute values, with the sign determined by the sign of the product of the numbers. This "product" is used to construct a vector product in R^N. The vector product induces the l_1 norm. The proposed additive neural network successfully solves the XOR problem. The experiments on MNIST dataset show that the classification performances of the proposed additive neural networks are very similar to the corresponding multi-layer perceptron and convolutional neural networks (LeNet).

乗算に代わってef-operationというものを導入してエネルギー効率のよいneural netを作ったという話。ef-operationとは符号は普通の乗算と同じだけど、絶対値は単なる和というもの。MNISTとCIFARで従来と同等の精度を出したという。


Lacking standardized extrinsic evaluation methods for vector representations of words, the NLP community has relied heavily on word similarity tasks as a proxy for intrinsic evaluation of word vectors. Word similarity evaluation, which correlates the distance between vectors and human judgments of semantic similarity is attractive, because it is computationally inexpensive and fast. In this paper we present several problems associated with the evaluation of word vectors on word similarity datasets, and summarize existing solutions. Our study suggests that the use of word similarity tasks for evaluation of word vectors is not sustainable and calls for further research on evaluation methods.

単語ベクトルの評価手法の一つである単語類似性タスクに関する問題点と既存の解決策を示したサーベイ論文。述べられている問題点は以下の7つ。 ①単語のrelatednessとsimilarityを区別していないデータセットが多い ②既存のデータセットではタスク固有のembeddingsを低く評価してしまう ③チューニングをテストセットで行って過適合させていることが多い ④単語類似性タスクの評価結果とNLPタスクに使用して評価した結果の相関性が低い ⑤評価結果に対する統計的有意性の欠如 ⑥コサイン類似度における頻度の影響 ⑦単語の多義性を考慮していない 結論としては単語ベクトルを評価するのに単語類似度タスクは適切ではないとしている。


The quality of word representations is frequently assessed using correlation with human judgements of word similarity. Here, we question whether such intrinsic evaluation can predict the merits of the representations for downstream tasks. We study the correlation between results on ten word similarity benchmarks and tagger performance on three standard sequence labeling tasks using a variety of word vectors induced from an unannotated corpus of 3.8 billion words, and demonstrate that most intrinsic evaluations are poor predictors of downstream performance. We argue that this issue can be traced in part to a failure to distinguish specific similarity from relatedness in intrinsic evaluation datasets. We make our evaluation tools openly available to facilitate further study.

単語表現の質は、単語類似性の人間の判断との相関を用いてよく評価される。しかし、そこでいい結果でも実際のタスクに適用すると良い結果にならないことを説明している。評価には、10の単語類似性の評価セットと3つのNLPタスク(POS, chunking, NER)を用いて、単語類似性の評価結果とNLPタスクの評価結果の相関を分析している。評価した結果、ほとんどのデータセットでは実際のNLPタスクの結果との間に負の相関があることがわかった。SimLex999だけ例外。このような結果となった原因として、ほとんどのデータセットでは単語の類似性と関連性を区別していないことによるものだと結論付けている。


Attention networks have proven to be an effective approach for embedding categorical inference within a deep neural network. However, for many tasks we may want to model richer structural dependencies without abandoning end-to-end training. In this work, we experiment with incorporating richer structural distributions, encoded using graphical models, within deep networks. We show that these structured attention networks are simple extensions of the basic attention procedure, and that they allow for extending attention beyond the standard soft-selection approach, such as attending to partial segmentations or to subtrees. We experiment with two different classes of structured attention networks: a linear-chain conditional random field and a graph-based parsing model, and describe how these models can be practically implemented as neural network layers. Experiments show that this approach is effective for incorporating structural biases, and structured attention networks outperform baseline attention models on a variety of synthetic and real tasks: tree transduction, neural machine translation, question answering, and natural language inference. We further find that models trained in this way learn interesting unsupervised hidden representations that generalize simple attention.

Attentionについて、単純にどの地点に注目するかというカテゴリカル分布的な考えでなく、規則性(構造)があると仮定しようという提案。具体的には、CRFの考えを用いてAttentionの候補となる「系列」に対して全体として最適になるように計算を行うという話。これで精度UPを確認。ただし、計算時間が遅くなるというハンデが伴うので注意。