the proposed approach nicely incorporates various ideas from recent work into a single meta-learning (or domain adaptation or incremental learning or ...) framework. although better empirical comparison to existing (however recent they are) approaches would have made it stronger, the reviewers all found this submission to be worth publication, with which i agree.

(optional) Comment on this decision.


ICLR 2018 Conference Acceptance Decision

Two reviewers recommended rejection, and the last reviewer votes for acceptance. The authors provided a rebuttal, including the end-to-end experiment (although the AC agrees with the authors that this experiment is not crucial to the paper). The AC read the paper and the reviews. While there are clearly interesting aspects of this work, it somewhat falls short in terms of the technical contribution. Perhaps a better writing would alleviate this issue: for example, explaining the visual features is somewhat a distraction from the main point, and could be put in the end. The 3 stage training is somewhat ad hoc (or less elegant). Since there are many excellent papers submitted to ICLR this year, this paper unfortunately did not make it above the bar.

How your identity will be displayed with the above content.

The users who will be allowed to read the above content.

(optional) Comment on this decision.

Accept (Oral)
Accept (Poster)
Reject
Invite to Workshop Track


quality: interesting idea to train an end-to-end attention together with CNNs and solid experiments to justify the benefits of using such attentions.
clarity: the presentation has been updated according to review comments and improved a lot
significance: highly relevant topic, good improvements over other methods

(optional) Comment on this decision.


The paper defines a new measure of influence and uses it to highlight important features. The definition is novel however, the reviewers have concerns regarding its significance, novelty and a thorough empirical comparison to existing literature is missing.

(optional) Comment on this decision.


The paper addresses the training time of CNNs, in the common setting where a CNN is trained on one domain and then used to extract features for another domain.  The paper proposes to speed up the CNN training step via a particular proposed training schedule with a reduced number of epochs.  Training time of the pre-trained CNN is not a huge concern, since this is only done once, but optimizing training schedules is a valid and interesting topic of study.   However, the approach here does not seem novel; it is typical to adjust training schedules according to the desired tradeoff between training time and performance.  The experimental validation is also thin, and the writing needs improvement.

(optional) Comment on this decision.


While this paper has some very interesting ideas the majority view of the reviewers and their aggregate numerical ratings are just too low to warrant acceptance.

(optional) Comment on this decision.


Dear authors,

After carefully reading the reviews, the rebuttal, and going through the paper, I regret to inform you that this paper does not meet the requirements for publication at ICLR.

While the variance analysis is definitely of interest, the reality of the algorithm does not match the claims. The theoretical rate is worse than that of SG but this could be an artefact of the analysis. Sadly, the experimental setup lacks in several ways:
- It is not yet clear whether escaping the saddle points is really an issue in deep learning as the loss function is still poorly understood.
- This analysis is done in the noiseless setting despite your argument being based around the variance of the gradients.
- You report the test error on CIFAR-10. While interesting and required for an ML paper, you introduce an optimization algorithm and so the quantity that matters the most is the speed at which you achieve a given training accuracy. Also, your table lists the value of the test accuracy rather than the speed of increase. Thus, you test the generalization ability of your algorithm while making claims about the optimization performance.

(optional) Comment on this decision.


The paper received three good quality reviews which were in agreement that the paper was below the acceptance threshold. The authors are encouraged to follow the suggestions from the reviews to revise the paper and resubmit to another venue.

(optional) Comment on this decision.


The paper studies end-to-end training of a multi-branch convolutional network. This appears to lead to strong accuracies on the CIFAR and SVHN datasets, but it remains unclear whether or not this results transfers to ImageNet. The proposed approach is hardly novel, and lacks a systematic comparison with "regular" ensembling methods and with related mixture-of-experts approaches (for instance: S. Gross et al. Hard Mixtures of Experts for Large Scale Weakly Supervised Vision, 2017; Shazeer et al. Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, 2017).

(optional) Comment on this decision.


The paper presents the Stein gradient estimator, a kernelized direct estimate of the score function for implicitly defined models. The authors demonstrate the estimator for GANs, meta-learning for approx. inference in Bayesian NNs, and approximating gradient-free MCMC. The reviewers found the method interesting and principled.  The GAN experiments are somewhat toy-ish as far as I am concerned, so I'd encourage the authors to try out larger-scale models if possible, but otherwise this should be an interesting addition to ICLR.

(optional) Comment on this decision.


This paper proposes improvements to WaveNet by showing that increasing connectivity provides superior models to increasing network size. The reviewers found both the mathematical treatment of the topic and the experiments to be of higher quality that most papers they reviewed, and were unanimous in recommending it for acceptance in the conference. I see no reason not to give it my strongest recommendation as well.

(optional) Comment on this decision.


I recommend acceptance based on the reviews. The paper makes novel contributions to learning one-hidden layer neural networks and designing new objective function with no bad local optima.

 There is one point that the paper is missing. It only mentions Janzamin et al in the passing. Janzamin et al propose using score function framework for designing alternative objective function. For the case of Gaussian input that this paper considers, the score function reduces to Hermite polynomials. Lack of discussion about this connection is weird. There should be proper acknowledgement of prior work. Also missing are some of the key papers on tensor decomposition and its analysis

I think there are enough contributions in the paper for acceptance irrespective of the above aspect.

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR. The paper presents an interesting analysis, but the utility of this analysis is questionable e.g. it is not clear how this might lead to improved VAEs/GANs. The authors did add an additional experimental result in their revised paper, but questions still remain. In light of this the significance of the paper is on the low side and it is therefore not ready for publication in ICLR without more work.

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR.  The consensus from the reviewers is that this is not quite ready for publication. There is also concern about whether ICLR, with its focus on representational learning, is the right venue for this work.

One of the reviewers initially submitted an incorrect review, but this mistake has now been rectified. Apologies that this was not done sooner in order to allow you to address their concerns.

(optional) Comment on this decision.


Meta score: 5

The paper explores an interesting idea, addressing a known bias in truncated BPTT by sampling across different truncated history lengths.  Limited theoretical analysis is presented along with PTB language modelling experimentation.   The experimental part could be stronger (e.g. trying to improve over the baseline) and perhaps more than just PTB.

Pros:
 - interesting idea
Cons:
 - limited analysis
 - limited experimentation

(optional) Comment on this decision.


The pros and cons of this paper cited by the reviewers can be summarized below:

Pros:
* Empirical results demonstrate decent improvements over other reasonable models
* The method is well engineered to the task

Cons:
* The paper is difficult to read due to grammar and formatting issues
* Experiments are also lacking detail and potentially difficult to reproduce
* Some of the experimental results are suspect in that the train/test accuracy are basically the same. Usually we would expect train to be much better in highly parameterized neural models
* The content is somewhat specialized to a particular task in NLP, and perhaps of less interest to the ICLR audience as a whole (although I realize that ICLR is attempting to cast a broad net so this alone is not a reason for rejection of the paper)

In addition to the Cons cited by the reviewers above, I would also note that there is some relevant work on morphology in sequence-to-sequence models, e.g.:
* "What do Neural Machine Translation Models Learn about Morphology?" Belinkov et al. ACL 2017.

and that it is common in sequence-to-sequence models to use sub-word units, which allows for better handling of morphological phenomena:
* "Neural Machine Translation of Rare Words with Subword Units" Sennrich et al. ACL 2016.

While the paper is not without merit, given that the cons seem to significantly outweigh the pros, I don't think that it is worthy of publication at ICLR at this time, although submission to a future conference (perhaps NLP conference) seems warranted.

(optional) Comment on this decision.


This paper and reviews makes for a difficult call.  The reviewers appear to be in agreement that Value Propagation provides an interesting algorithmic advance over earlier work on Value Iteration networks.  AnonReviewer1 gives a strong rationale why the advance is both original and significant.  Their experiments also show very nice results with VProp and MVProp in 2-D grid-worlds.

However, I also fully agree with AnonReviewer2 that testing in other domains beyond 2-D grid-world is necessary.  Earlier work on VIN was also tested on a Mars Rover / continuous control domain, as well as graph-based web navigation task.  The authors' rebuttal on this point comes across as weak.  In their view, they can't tackle real-world domains until VProp has been proven effective in large, complex grid-worlds.  I don't buy this at all -- they could start initial experiments right away, which would perhaps yield some surprising results. Given this analysis, the committee recomments this paper for workshop.

Pros: significant algorithmic advance, good technical quality and writeup, nice results in 2-D grid world.

Con: Validation is only in 2-D grid-world domains.

(optional) Comment on this decision.


This paper describes a method to generate provably 'optimal' adversarial examples, leveraging the so-called 'Reluplex' technique, which can evaluate properties of piece-wise linear representations.
Reviewers agreed that incorporating optimality certificates into adversarial examples is a promising direction to follow, but were also concerned about the lack of empirical justification the current paper provides and missed discussion about the relevance of choosing Lp distances. They all recommended pushing experiments to more challenging datasets before the paper can be accepted, and the AC shares the same advice.

(optional) Comment on this decision.


The paper provides a constrained mutual information objective function whose Lagrangian dual covers several existing generative models. However reviewers are not convinced of the significance or usefulness of the proposed unifying framework (at least from the way results are presented currently in the paper). Authors have not taken any steps towards revising the paper to address these concerns. Improving the presentation to bring out the significance/utility of the proposed unifying framework is needed.

(optional) Comment on this decision.


The paper received mixed reviews with scores of 5 (R1), 5 (R2),  7 (R3).  All three reviewers raise concerns about the lack of comparisons to other methods. The rebuttal is not compelling on this point. There are quite a few methods that could be used for this application available (often with source code) and should be compared to, e.g. DenseNets (Huang et al.). Given that the proposed method isn't in of itself hugely novel, a thorough experimental evaluation is crucial to the justifying the approach. The AC has closely looked at the rebuttal and the paper and feels that it cannot be accepted for this reason at this time.

(optional) Comment on this decision.


The paper proposes a neural net based method for active localization in a known map using a learnt perception model (convnet) and a learnt control policy combined with a set belief state representation. The method compares well to baselines and has good accuracy in 2d and 3d envs. All three reviewers are in favor of acceptance due to the novelty and competitive performance of the approach.

(optional) Comment on this decision.


The paper presents a boosting method and uses it to train an ensemble of convnets for image classification. The paper lacks conceptual and empirical comparisons with alternative boosting and ensembling methods. In fact, it is not even clear from the experimental results whether or not the proposed method outperforms a simple baseline model that averages the predictions of T independently trained convolutional networks.

(optional) Comment on this decision.


Reviews are marginal.
I concur with the two less-favorable reviews that the metrics  for privacy protection are not sufficiently strong for preserving privacy.

(optional) Comment on this decision.


This paper provides an interesting synthesis of ideas. Although the results could be improved, this is a good paper.

(optional) Comment on this decision.


This paper is interesting since it goes to showing the role of model averaging. The clarifications made improve the paper, but the impact of the paper is still not realised: the common confusion on the retraining can be re-examined, clarifications in the methodology and evaluation, and deeper contextulaisation of the wider literature.

(optional) Comment on this decision.


Regarding clarity, while the paper definitely needs work if it is to be resubmitted to an ML venue, different revisions would be appropriate for a physics audience. And given the above comment, any suggested changes are likely to be superfluous.

(optional) Comment on this decision.


The reviewers (all experts in this area) appreciated the novelty of the idea, though they felt that the experimental results (samples and Inception scores) did not provide convincing evidence value of this method over already established techniques. The authors responded to the concerns but were not able to address the issue of evaluation due to time constraints. The idea is likely sound but evaluation does not meet the bar, it may make a good contribution as a workshop paper.

(optional) Comment on this decision.


Authors present a method for representing DNA sequence reads as one-hot encoded vectors, with genomic context (expected original human sequence), read sequence, and CIGAR string (match operation encoding) concatenated as a single input into the framework. Method is developed on 5 lung cancer patients and 4 melanoma patients.

Pros:
- The approach to feature encoding and network construction for task seems new.
- The target task is important and may carry significant benefit for healthcare and disease screening.

Cons:
- The number of patients involved in the study is exceedingly small. Though many samples were drawn from these patients, pattern discovery may not be generalizable across larger populations. Though the difficulty in acquiring this type of data is noted.
- (Significant) Reviewer asked for use of public benchmark dataset, for which authors have declined to use since the benchmark was not targeted toward task of ultra-low VAFs. However, perhaps authors could have sourced genetic data from these recommended public repositories to create synthetic scenarios, which would enable the broader research community to directly compare against the methods presented here. The use of only private datasets is concerning regarding the future impact of this work.
- (Significant) The concatenation of the rows is slightly confusing. It is unclear why these were concatenated along the column dimension, rather than being input as multiple channels. This question doesn't seem to be addressed in the paper.
Given the pros and cons, the commitee recommends this interesting paper for workshop.

(optional) Comment on this decision.


This paper presents a distributed memory architecture based on a generative model with a VAE-like training criterion. The claim is that this approach is easier to train than other memory-based architectures. The model seems sound, and it is described clearly. The experimental validation seems a bit limited: most of the comparisons are against plain VAEs, which aren't a memory-based architecture. The discussion of "one-shot generalization" is confusing, since the task is modified without justification to have many categories and samples per category. The experiment of Section 4.4 seems promising, but this needs to be expanded to more tasks and baselines since it's the only experiment that really tests the Kanerva Machine as a memory architecture. Despite these concerns, I think the idea is promising and this paper contributes usefully to the discussion, so I recommend acceptance.

(optional) Comment on this decision.


The reviewers have various reservations.
While the paper has interesting suggestions, it is slightly incremental and the results are not sufficiently compared to other techniques.
We not that one reviewer revised his opinion

(optional) Comment on this decision.


This paper attempts to decouple two factors underlying the success of GANs: the inductive bias of deep CNNs and adversarial training. It shows that, surprisingly, the second factor is not essential. R1 thought that comparisons to Generative Moment Matching Networks and Variational Autoencoders should be provided (note: this was added to a revised version of the paper). They also pointed out that the paper lacked comparisons to newer flavors of GANs. While R1 pointed out that the use of 128x128 and 64x64 images was weak, I tend to disagree as this is still common for many GAN papers. R2 was neutral to positive about the paper and thought that most importantly, the training procedure was novel. R3 also gave a neutral to positive review, claiming the paper was easy to follow and interesting. Like R1, R3 thought that a stronger claim could be made by using different datasets. In the rebuttal, the authors argued that the main point was not in proposing a state-of-the-art generative model of images but to provide more an introspection on the success of GANs. Overall, I found the work interesting but felt that the paper could go through one more review/revision cycle. In particular, it was very long. Without a champion, this paper did not make the cut.

(optional) Comment on this decision.


This paper studies the problem of modeling logical structure in a neural model.  It introduces a data set for probing various existing models and proposes a new model that addresses shortcomings in existing ones.  The reviewers point out that there is a bit of a tautology in introducing a new task and a new model that solves it.  The revised version addresses some of those concerns.  Overall, it is a thought-provoking and well-written study that will be interesting to discuss at ICLR.

(optional) Comment on this decision.


An interesting model, for an interesting problem but perhaps of limited applicability - doesn't achieve state of the art results on practical tasks.
Paper has other limitations, though the authors have addressed some in rebuttals.

(optional) Comment on this decision.


Reviewers are unanimous in scoring this paper below threshold for acceptance.  The authors did not submit any rebuttals of the reviews.

Pros:
Paper is generally clear.
Hardware results are valuable.

Cons:
Limited simulation results.
Proposed method is not really novel.
Insufficient empirical validation of the approach.

(optional) Comment on this decision.


Differentiable neural networks used as a measure of design optimality in order to improve efficiency of automated design.

Pros:
- Genetic algorithms, which are the dominant optimization routine for automated design systems, can be computationally expensive. This approach alleviates this bottleneck under certain circumstances and applications.

Cons:
- Primarily application paper, machine learning advancement is marginal.
- Multiple reviewers: Generalization capability not clear. For example, some utility systems may be stochastic (i.e. turbulence) and require multiple trials to measure fitness, which this method would not be able to model.
Overall, the committee feels this paper is interesting enough to appears as a workshop paper.

(optional) Comment on this decision.


All reviewers are unanimous that the paper is below threshold for acceptance.  The authors have not provided rebuttals, but merely perfunctory generic responses.

I think the most important criticism is that the approach is "very ad-hoc."  I would encourage the authors to consider more principled ways of automatically designing reward functions, like for example, Inverse Reinforcement Learning, in which you start with a good agent behavior policy, and then estimate a reward function for which the behavior policy maximizes the reward function.

(optional) Comment on this decision.


Generally solid engineering work but a bit lacking in terms of novelty and some issues with clarity. At the end of the day the empirical gains are not sufficient for acceptance - the results are state-of-the-art relative to published work, but not in the top 10 based on the official leaderboard (not even at time of submission). Since the technical contributions are small and the engineering contributions have been made obsolete by concurrent work, I suggest rejection.

(optional) Comment on this decision.


Meta score: 6

This is a thorough empirical paper, demonstrating the effectiveness of a relatively simple model for recommendations:

Pros:
 - strong experiments
 - always good to see simple models pushed to perform well
 - presumably of interest to practioners in the area

Cons:
 - quite oriented to the recommendation application
 - technical novelty is in the experimental evaluation rather than any new techniques

On balance I recommend the paper is invited to the workshop.

(optional) Comment on this decision.


The paper proposes a new sparse matrix representation based on Viterbi algorithm with high and fixed index compression ratio regardless of the pruning rate.  The method allows for faster parallel decoding and achieves improved compression of index data storage requirement over existing methods (e.g., magnitude-based pruning) while maintaining the pruning rate. The quality of paper seems solid and of interest to a subset of the ICLR audience.

(optional) Comment on this decision.


Meta score: 7

This paper presents a novel architecture for neural network based TTS using a memory buffer architecture.  The authors have made good efforts to evaluate this system against other state-of-the-art neural TTS systems, although this is hampered by the need for re-implementation and the evident lack of optimal hyperparameters for e.g. tacotron.  TTS is hard to evaluate against existing approaches, since it requires subjective user evaluation.  But overall, despite its limtations, this is a good and interesting paper which I would like to see accepted

Pros:
 - novel architecture
 - good experimentation on multiple databases
 - good response to reviewer comments
 - good results

Cons:
 - some problems with the experimental comparison (baselines compared against)
 - writing could be clearer, and sometimes it feels like the authors are slightly overclaiming

I take  the point that this might be more suitable for a speech conference, but it seems to me that paper offers enough to the ICLR community for it to be worth accepting.

(optional) Comment on this decision.


All reviewers agree that the proposed method is novel and experiments do a good job in establishing its value for few-shot learning. Most the concerns raised by the reviewers on experimental protocols have been addressed in the author response and revised version.

(optional) Comment on this decision.


PROS:
1. Clear, interesting idea.
2. Largely convincing evaluation
3. Good writing

CONS:
1. The model used in the evaluation is a Resnet-50 and could have been more convincing with a more SOTA model.
2. There is some concern about the whether the comparison of results (fig 6c) is really apples to apples.

(optional) Comment on this decision.


here, yet another sentence representation method is proposed. i agree with R1 and R3 that this does not contribute significantly to be a full-length conference paper.

(optional) Comment on this decision.


The pros and cons of the paper can be summarized below:

Pro:
* The improvements afforded by the method are significant over baselines, although these baselines are very preliminary baselines on a new dataset.

Con
* There is already a significant amount of work in using grammars to guide semantic parsing or code generation, as rightfully noted by the authors, and thus the approach in the paper is not extremely novel.
* Because there is no empirical comparison with these methods, the relative utility of the proposed method is not clear.

As a result, I recommend that the paper not be accepted at this time.

(optional) Comment on this decision.


Two reviewers recommended rejection, and one is slightly more positive. The main concern is that the experiments are not convincing (ie, the number of base and added classes is very small). Furthermore, while the paper introduces several interesting ideas, the AC agrees with the second reviewer that each of these could be explored in more detail. This work seems preliminary. The authors are encouraged to resubmit to a future conference.

(optional) Comment on this decision.


This is an interesting paper, but was quite difficult to follow. As they stand, the empirical results are not altogether convincing nor warrant acceptance.

(optional) Comment on this decision.


All of the reviewers agree that the experimental results are promising and the proposed activation function enables a decent degree of quantization. However, the main concern with the approach is its limited novelty compared to previous work on clipped activation functions.

minor comments:
- Even though PACT is very similar to Relu, the names are very different.
- Please include a plot showing the proposed activation function as well.

(optional) Comment on this decision.


This paper presents a MIL method for medical time series data. General consensus among reviewers that work does not meet criteria for being accepted.

Specifically:

Pros:
- A variety of meta-learning parameters are evaluated for the task at hand.
- Minor novelty of the proposed method

Cons:
- Minor novelty of the proposed method
- Rationale behind architectural design
- Thoroughness of experimentation
- Suboptimal choice of baseline methods
- Lack of broad evaluation across applications for new design
- Small dataset size
- Significance of improvement

(optional) Comment on this decision.


This paper does not meet the acceptance bar this year, and thus I must recommend it for rejection.

(optional) Comment on this decision.


This paper proposes an interesting machinery around Generative Adversarial Networks to enable sampling not only from conditional observational distributions but also from interventional distributions. This is an important contribution as this means that we can obtain samples with desired properties that may not be present in the training set; useful in applications such as ones involving fairness and also when data collection is expensive and biased. The main component called the causal controller models the label dependencies and drives the standard conditional GAN. As reviewers point out, the causal controller assumes the knowledge of the causal graph which is a limitation as this is not known a priori in many applications. Nevertheless, this is a strong paper that convincingly demonstrates a novel approach to incorporate causal structure into generative models. This should be of great interest to the community and may lead to interesting applications that exploit causality. I recommend acceptance.

(optional) Comment on this decision.


Overall, the paper is missing a couple of ingredients that would put it over the bar for acceptance:

- I am mystified by statements such as "RL2 no longer gets the best final performance." from one revision to another, as I have lower confidence in the results now.

- More importantly, the paper is missing comparisons of the proposed methods on *already existing* benchmarks. I agree with Reviewer 1 that a paper that only compares on benchmarks introduced in the very same submission is not as strong as it could be.

In general, the idea seems interesting and compelling enough (at least on the Krazy World & maze environments) that I can recommend inviting to the workshop track.

(optional) Comment on this decision.


The paper received borderline-negative reviews with scores of 5,5,6. A consistent issue was the weakness of the experiments: (i) lack of comparison to appropriate baselines, (ii) differences between published/reported numbers for DeepLab-ResNet (R3) and (iii) related work, e.g. Wojna paper, as raised by R1. The AC did not find the author's responses to these issues convincing. For (ii) the gap between 73 and 79 is large and the author's explanation for the difference doesn't seem plausible. For (iii), the response promised comparisons/discussion but there were not added to the draft.

Given this, the paper cannot be accepted in it current form. The experiments should be improved before the paper is resubmitted.

(optional) Comment on this decision.


meta score: 4

The paper has been extensively edited during the review process - the edits are so extensive that I think the paper requires a re-review, which is not possible for ICLR 2018

Pros:
 - potentially interesting and novel approach to prefix encoding for character level CNN text classification
 - some experimental comparisons
Cons:
 - lacks good comparison with the state-of-the-art, which makes it difficult to determine conclusions
 - writing style lacks clarity.

I would recommend that the authors continue to improve the paper and submit it to a later conference.

(optional) Comment on this decision.


The key concern from the reviewers that was not addressed is that none of the experimental results illustrate convergence vs. time instead of convergence vs. number of iterations.  While the authors point out that their method is O(ND) instead of O(KND), the reviewers really wanted to see graphs demonstrating this, given that the implicit SGD method requires an iterative solver. The revised paper is otherwise much improved from the original submission, but falls a bit short of ICLR acceptance because of the lack of a measurement of convergence vs. time.

Pros:
+ Promising unbiased algorithms for optimizing the log-likelihood of a model using a softmax without having to repeatedly compute the normalizing factor.

Cons:
- The key concern from the reviewers that was not addressed is that none of the experimental results illustrate convergence vs. time instead of convergence vs. number of iterations.

(optional) Comment on this decision.


The paper proposes a method for learning connectivity in neural networks, evaluated on the ResNeXt architecture. The novelty of the method is rather limited, and even though the method has been shown to improve on the ResNeXt baselines on CIFAR-100 and ImageNet classification tasks (which is encouraging), it should have been evaluated on more architectures and datasets to confirm its generality.

(optional) Comment on this decision.


This paper presents yet another scheme for weight tying for compressing neural networks, which looks a lot like a less Bayesian version of recent related work, and gets good empirical results on realistic problems.

This paper is well-executed and is a good contribution, but falls below the bar on
1) Discovering something new and surprising, except that this particular method (which is nice and simple and sensible) works well.  That is, it doesn't advance the conversation or open up new directions.
2) Potential impact (although it might be industrially relevant)

Also, the title is a bit overly broad given the amount of similar existing work.

(optional) Comment on this decision.


This is a good contribution, with the potential to become extremely good and significant if presentation is substantially improved.
All reviewers comment on the lack of clarity of the paper, especially concerning its central contributions (Section 4 and 5), as illustrated also by the relatively low confidence scores.
Reviewers also mention the current imbalance between the generality of high-order compositional networks and the motivation and empirical evaluation of these models. Generalizations of graph neural representations based on higher order local interactions are particularly interesting in contexts such as combinatorial optimization, where heuristics typically exploit high-order interactions.

In summary, we believe this work deserves a further iteration before it can be in proceedings in order to improve the exposition and the motivation of compositional networks, that will greatly improve its exposure to the community.  That said, the idea it lays forward is of potential interest, and thus the AC recommends resubmission to the workshop track.

(optional) Comment on this decision.


This paper tackles a very important problem: evaluating natural language generation. The paper presents an overview of existing unsupervised metrics, and looks at how they correlate with human evaluation scores. This is important work and the empirical conclusions are useful to the community, but the datasets used are too limited and the authors agree it would be better to use newer bigger and more diverse datasets suggested by reviewers for drawing more general conclusions. This work would indeed be much stronger if it relied on better, more recent datasets; therefore publication as is seems premature.

(optional) Comment on this decision.


This paper proposes a theoretically-motivated method for combining reinforcement learning and imitation learning. There was some disagreement amongst the reviewers, but the AC was satisfied with the authors' rebuttal.

(optional) Comment on this decision.


Some reviewers seem to assign novelty to the compression and classification formulation; however, semi-supervised autoencoders have been used for a long time. Taking the compression task more seriously as is done in this paper is less explored.

The paper provides some extensive experimental evaluation and was edited to make the paper more concise at the request of reviewers. One reviewer had a particularly strong positive rating, due to the quality of the presentation, experiments and discussion. I think the community would like this work and it should be accepted.

(optional) Comment on this decision.


Below is a summary of the pros and cons of the proposed paper:

Pros:
* Proposes a novel method to tune program synthesizers to generate correct programs and prune search space, leading to better and more efficient synthesis
* Shows small but substantial gains on a standard benchmark

Cons:
* Reviewers and commenters cited a few clarity issues, although these have mostly been resolved
* Lack of empirical comparison with relevant previous work (e.g. Parisotto et al.) makes it hard to determine their relative merit

Overall, this seems to be a solid, well-evaluated contribution and seems to me to warrant a poster presentation.

Also, just a few notes from the area chair to potentially make the final version better:

The proposed method is certainly different from the method of Parisotto et al., but it is attempting to solve the same problem: the lack of consideration of the grammar in neural program synthesis models. The relative merit is stated to be that the proposed method can be used when there is no grammar specification, but the model of Parisotto et al. also learns expansion rules from data, so no explicit grammar specification is necessary (as long as a parser exists, which is presumably necessary to perform the syntax checking that is core to the proposed method). It would have been ideal to see an empirical comparison between the two methods, but this is obviously a lot of work. It would be nice to have the method acknowledged more prominently in the description, perhaps in the introduction, however.

It is nice to see a head-nod to Guu et al.'s work on semantic parsing (as semantic parsing from natural language is also highly relevant). There is obviously a lot of work on generating structured representations from natural lanugage, and the following two might be particularly relevant given their focus on grammar-based formalisms for code synthesis from natural language:

* "A Syntactic Neural Model for General-purpose Code Generation" Yin and Neubig ACL 2017.
* "Abstract Syntax Networks for Code Generation and Semantic Parsing" Rabinovich et al. ACL 2017

(optional) Comment on this decision.


This paper proposes to use data-driven deep convolutional architectures for modeling advection diffusion. It is well motivated and comes with convincing numerical experiments.
Reviewers agreed that this is a worthy contribution to ICLR with the potential to trigger further research in the interplay between deep learning and physics.

(optional) Comment on this decision.


The experimental work in this paper leaves it just short of being suitable for acceptance.
The work needs more comparisons with prior work and other approaches.
The numerical ratings of the work by reviewers are just too low.

(optional) Comment on this decision.


This work shows how activation patterns of units reminiscent of grid and border cells emerge in RNNs trained on navigation tasks. While the ICLR audience is not mainly focused on neuroscience, the findings of the paper are quite intriguing, and grid cells are sufficiently well-known and "mainstream" that this may interest many people.

(optional) Comment on this decision.


The paper proposes a novel approach for DNN inversion mainly targeted towards semi-supervised learning. However the semi-supervised learning results are not competitive enough. Although the authors mention in the author-response that semi-supervised learning is not the main goal of the paper, the experiments and claims of the paper are mainly targeted towards semi-supervised learning. As the approach for inversion is novel, the paper could be motivated from a different angle with appropriate supporting experiments. In its current form it's not suitable for publication.

(optional) Comment on this decision.


This paper with the self-explanatory title was well received by the reviewers and, additionally, comes with available code. The paper builds on prior work (Sinkhorn operator) but shows additional, significant amount of work to enable its application and inference in neural networks.  There were no major criticisms by the reviewers, other than obvious directions for improvement which should have been already incorporated in the paper, issues with clarity and a little more experimentation. To some extent, the authors addressed the issues in the revised version.

(optional) Comment on this decision.


The pros and cons of this paper cited by the reviewers can be summarized below:

Pros:
* Solid experimental results against strong baselines on a task of great interest
* Method presented is appropriate for the task
* Paper is presented relatively clearly, especially after revision

Cons:
* The paper is somewhat incremental. The basic idea of aggregating across multiple examples was presented in Kadlec et al. 2016, but the methodology here is different.

(optional) Comment on this decision.


Meta-score: 4

This paper presents an approach which uses attention across multiple speech or video channels.  After some synthetic experiments, presents experiments on chime-3, but has a rather weak baseline system

Pros:
 - addresses an interesting task

Cons:
 - does not take account of other recent papers in the area
 - experimental results are weak - very high errors in baseline system
 - limited novelty

(optional) Comment on this decision.


A novel dual memory system inspired by brain for the important incremental learning and very good results.

(optional) Comment on this decision.


The paper presents a modified sampling method for improving the quality of interpolated samples in deep generative models.

There is not a great amount of technical contributions in the paper, however it is written in a very clear way, makes interesting observations and analyses and shows promising results. Therefore, it should be of interest to the ICLR community.

(optional) Comment on this decision.


This paper proposes a way to automatically weight different tasks in a multi-task setting.  The problem is a bit niche, and the paper had a lot of problems with clarity, as well as the motivation for the experimental setup and evaluation.

(optional) Comment on this decision.


The paper is well written overall. However, the algorithmic framework has limited novelty and the reviewers unanimously are unconvinced by experimental results showing marginal improvements on smallish UCI datasets.

(optional) Comment on this decision.


This paper presents a nice set of results on a new RL algorithm. The main downside is the limitation to the Atari domain, but otherwise the ablation studies are nice and the results are strong.

(optional) Comment on this decision.


This paper investigates the performance of various second-order optimization methods for training neural networks. Comparing different optimizers is worthwhile, but as this is an empirical paper which doesn't present novel techniques, the bar is very high for the experimental methodology. Unfortunately, I don't think this paper clears the bar: as pointed out by the reviewers, the comparisons miss several important methods, and the experiments miss out on important aspects of the comparison (e.g. wall clock time, generalization). I don't think there is enough of a contribution here to merit publication at ICLR, though it could become a strong submission if the reviewers' points were adequately addressed.

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR. The paper studies an interesting problem and the solution, which fuses student-teacher approaches to continual learning and variational auto-encoders, is interesting. The revision of the paper has improved readability. However, although the framework is flexible, it is complex and appears rather ad hoc as currently presented. Exploration of the effect of the many hyper-parameters or some more supporting theoretical work / justification would help. The experimental comparisons were varied, but adding more baselines e.g. comparing to a parameter regularisation approach like EWC or synaptic intelligence applied to a standard VAE would have been enlightening.

Summary: There is the basis of a good paper here, but a comprehensive experimental evaluation of design choices or supporting theory would be useful for assessing what is a complex approach.

(optional) Comment on this decision.


Mixed precision application of CNNs is being explored for e.g. hardware implementations of networks trained at full precision.  Mixed precision at training time is less common.  This submission primarily concerns itself with the practical implementation details of training with mixed precision, and focuses primarily on representation of mixed precision floating point and algorithmic issues for learning.  In the end the support for the approach is primarily empirical, with the mixed precision approach giving a factor of two speedup with half the precision, while accuracies remain effectively statistically tied on the ImageNet 1k database.  Table 1 should avoid the use of bold as there is likely no statistical significance.

The reviewers appreciated the paper. The proposed approach is sensible, and appears correct.

(optional) Comment on this decision.


This paper does not meet the bar for ICLR - neither in terms of the quality of the write-up, nor in experimental design. The two confident reviewers agree to reject the paper, the weak accept comes from a less confident reviewer who did not write a good review at all. The rebuttal does not change this assessment.

(optional) Comment on this decision.


Understanding the generalization behavior of deep networks is certainly an open problem. While this paper appears to develop some interesting new Fourier-based methods in this direction, the analysis in its current form is currently too restrictive, with somewhat limited empirical support, to broadly appeal to the ICLR community. Please see the reviews for more details.

(optional) Comment on this decision.


Though the approach is not terribly novel, it is quite effective (as confirmed on a wide range of evaluation tasks). The approach is simple and likely to be useful in applications. The paper is well written.

+ simple and efficient
+ high quality evaluation
+ strong results
- novelty is somewhat limited

(optional) Comment on this decision.


Understanding the quality of the solutions found by gradient descent for optimizing deep nets is certainly an important area of research. The reviewers found several intermediate results to be interesting.  At the same time, the reviewers unanimously have pointed out various technical aspects of the paper that are unclear, particularly new contributions relative to recent prior work. As such, at this time, the paper is not ready for ICLR-2018 acceptance.

(optional) Comment on this decision.


The idea of extending deep nets to infinite dimensional inputs is interesting but, as the reviewers noted, the execution does not have the quality we can expect from an ICLR publication. I encourage the authors to consider the meaningful comments that were made and modify the paper accordingly.

(optional) Comment on this decision.


Overall this paper seems to make an interesting contribution to the problem of subtask discovery, but unfortunately this only works in a tabular setting, which is quite limiting.

(optional) Comment on this decision.


The paper presents a variant of network morphism (Wei et al., 2016) for dynamically growing deep neural networks. There are some novel contributions (such as OptGD for finding a morphism given the parent network layer). However, in the current form, the experiments mostly focus on comparisons against fixed network structure (but this doesn't seem like a strong baseline, given Wei et al.'s work), so the paper should provide more comparisons against Wei et al. (2016) to highlight the contribution of this work. In addition, the results will be more convincing if the state-of-the-art performance can be demonstrated for large-scale problems (such as ImageNet classification).

(optional) Comment on this decision.


meta score: 4
This paper concerns a variant to previous RNN architectures using temporal skip connections, with experimentation on the PTB language modelling task
The reviewers all recommend that the paper is not ready for publication and thus should be rejected from ICLR.  The novelty of the paper and its relation to the state-of-the-art is not clear.  The experimental validation is weak.
Pros:
 - possibly interesting idea
Cons:
 - weak experimental validation
 - weak connection to the state of the art
 - precise original contribution w.r.t state-of-the-art is not clear

(optional) Comment on this decision.


The paper proposes interesting  deep learning based spectral clustering techniques. The use of functional embeddings for enabling spectral clustering to have an out-of-sample extension has of course been explored earlier (e.g., see Manifold Regularization work of Belkin et al, JMLR 2006). For polynomials or kernel-based spectral clustering, the orthogonality of the outputs can be exactly handled via a generalized eigenvector problem, while here the arguments are statistically flavored and not made very clear in the original draft. Some crucial comparisons, e.g., against large-scale versions of vanilla spectral clustering and against other methods that generalize to new samples is missing or not thorough enough. See reviews for more precise description of issues. As such the paper will benefit from a revision.

(optional) Comment on this decision.


This paper provides an important discussion about the relationship between training efficiency and label redundancy. The updates to the paper will improve the paper further. Reviewers found the paper interesting, well written, and addresses and important problem.

(optional) Comment on this decision.


The reviewers agree that the problem being studied is important and relevant but express serious concerns. I recommend the authors to carefully go through the reviews and significantly scale up their experiments.

(optional) Comment on this decision.


This paper addresses an important application in genomics, i.e. the prediction of chromatin structure from nucleotide sequences.  The authors develop a novel method for converting the nucleotide sequences to a 2D structure that allows a CNN to detect interactions between distant parts of the sequence.  The reviewers found the paper innovative, interesting and convincing.  Two reviewers gave a 7 and there was one 6.  The 6, however, indicated during rather lengthy discussion that they were willing to raise their scores if their comments were addressed.  Hopefully the authors will address these comments in the camera ready version.  Overall a solid application paper with novel insights and technical innovation.

(optional) Comment on this decision.


The reviewers agreed that the work addresses an important problem. There was disagreement as to the correctness of the arguments in the paper: one of these reviewers was eventually convinced. The other pointed out another two issue in their final post, but it seems that 1. the first is easily adopted and does not affect the correctness of the experiments and 2. the second was fixed in the second revision. Ideally these would be rechecked by the third reviewer, but ultimately the correctness of the work is the authors' responsibility.

Some related work (by McAllister) was pointed out late in the process. I encourage the authors to take this related work seriously in any revisions. It deserves more than two sentences.

(optional) Comment on this decision.


+ clearly written and thorough empirical comparison of several metrics/divergences for evaluating GANs, prominently parametric-critic based divergences.
 - little technical novelty with respect to prior work. As noted by reviewers and an anonymous commentator:  using an Independent critic for evaluation has been proposed and used in practice before.
 +  the contribution of the work thus lies primarily in its well-done and extensive empirical comparisons of multiple metrics and models

(optional) Comment on this decision.


The key motivation for the work is producing both an efficient (parallelizable / fast) and accurate reading comprehension model. At least two reviewers are not convinced that this goal is really achieved (e.g., no comparison to hierarchical modeling, performance is not as strong).   I also share concerns of R1 that, without proper ablation search and more careful architecture choice, the modeling decisions seem somewhat arbitrary.

+ the goal (of achieving effective reading comprehesion models) is important
- alternative parallelization techniques (e.g., hierarchical modeling) are not considered
- ablation studies / more systematic architecture search are missing
- it is not clear that the drop in accuracy can be justified by the potential efficiency gains (also see details in R3 -> no author response to them)

(optional) Comment on this decision.


The authors make an empirical study of the "dimension" of a neural net optimization problem, where the "dimension" is defined by the minimal random linear parameter subspace dimension where a (near) solution to the problem is likely to be found.   I agree with reviewers that in light of the authors' revisions, the results are interesting enough to be presented at the conference.

(optional) Comment on this decision.


The reviewers agree that the paper studies and interesting problem with an interesting approach. The reviewers raised some concerns regarding the theoretical and empirical results. The authors have made changes to the paper, but given the theoretical nature of the paper and the extent of changes, another review is needed before publication.

(optional) Comment on this decision.


The reviewers agreed that the paper was somewhat preliminary in terms of the exposition and empirical work.  They all find the underlying problem quite interesting and challenging (i.e. spiking recurrent networks).  However, the manuscript failed to motivate the approach.  In particular, everyone agrees that spiking networks are very interesting, but it's unclear what problem the presented work is solving.  The authors need to be more clear about their motivation and then close the loop with empirical validation that their approach is solving the motivating problem (i.e. do we learn something about biological plausibility, are spiking networks better than traditional LSTMs at modeling a particular kind of data, or are they more efficiently implemented on hardware?).  Motivating the work with one of these followed by convincing experiments would make this a much stronger paper.

Pros:
- Tackles an interesting and challenging problem at the intersection of neuroscience and ML
- A novel method for creating a spiking LSTM

Cons:
- The motivation is not entirely clear
- The empirical analysis is too simple and does not demonstrate the advantages of this approach
- The paper seems unfocused and could use rewriting

(optional) Comment on this decision.


PROS:
1. good results; the authors made it work
2. paper is largely well written

CONS:
1. some found the writing to be unclear and sloppy in places
2. the algorithm is complicated -- a chain of sub-algorithms

A few small points:

-I initially found Algorithm 1 to be confusing because it wasn't clear whether it was intended to be invoked for each task (making the training depend on all the datasets).  I finally convinced myself that this was not the intention and that the inner loop of the algorithm is what is actually executed incrementally.

(optional) Comment on this decision.


Paper presents and interesting new direction, but the evaluation leaves many questions open, and situation with respect to state of the art is lacking

(optional) Comment on this decision.


This paper is an easy accept -- three reviewers have above threshold scores, while one reviewer is slightly below threshold, but based on the submitted manuscript.  It appears that the paper has substantially improved based on reviewer comments.

Pros:

All reviews had positive sentiment: "very elegant and general idea" (Reviewer4); "idea is interesting and potentially very useful" (Reviewer2); "method is novel, the explanation is clear, and has good experimental results" (Reviewer3); "a good way to learn a policy for resetting while learning a policy for solving the problem.  Seems like a fairly small but well considered and executed piece of work." (Reviewer1)

Cons:

One reviewer found that testing in only three artificial tasks was a limitation.

The initial reviews noted several issues where clarification of the text and/or figures was needed.  There were also a bunch of statements where the reviewers questioned the technical correctness / accuracy of the discussion.  Most of these points appear to have been adequately addressed in the revised manuscript.

(optional) Comment on this decision.


This work extends Druckmann and Chklowskii, 2012 and demonstrates some interesting properties of the new model. This would be of interest to a neuroscience audience, but the focus is off for ICLR.

(optional) Comment on this decision.


This paper extends work on neural architecture search by introducing a new framework for searching and experiments on new domains of NMT and QA. The results of the work are  beneficial and show improvements using this approach. However the reviewers point out significant issues with the approach itself:

- There is skepticism about the use of NAS in general, particular compared to using the same computational power for other types of simpler hyperparameter search.
- There is general concern about the use of such large scale brute force methods in general. Several of the reviewers expressed concerns about ever possibly being able to replicate these results.
- Given the computational power required, the reviewers feel like the gains are not particularly large, for instance the Squad results not being compared to the best reported systems.

(optional) Comment on this decision.


The reviewers are not convinced by a number of aspects: including originality and clarity. Whereas the assessment of clarity and originality may be somewhat subjective (though the connections between margin-based loss and negative sampling is indeed well known), it is pretty clear that evaluation is very questionable. This is not so much about existence of more powerful factorizations  (e.g., ConvE / HolE) but the fact that the shown baselines (e.g., DistMult) can be tuned to yield much better performance on these benchmarks.  Also, indeed the authors should report results on cleaned versions of the datasets (e.g., FB15k-237).  Overall, there is a consensus that the work is not ready for publication.

Pros:
-- In principle, new insights on standardly used methods would have been very interesting

Cons:
-- Evaluation is highly problematic
-- At least some results do not seem so novel / interesting; there are questions about the rest (e.g., assumptions)
-- The main advantage of sq loss methods is that it enables the alternating least squares algorithm, does not seem possible here (at least not shown)

(optional) Comment on this decision.


This paper clearly surveys a set of methods related to using generative models to produce samples with desired characteristics.  It explores several approaches and extensions to the standard recipe to try to address some weaknesses.  It also demonstrates a wide variety of tasks.  The exposition and figures are well-done.

(optional) Comment on this decision.


This is a fascinating paper, and representative of the sort of work which is welcome in our field and in our community. It presents a compiler framework for the development of DSLs (and models) for Deep Learning and related methods. Overall, reviewers were supportive of and excited by this line of work, but questioned its suitability for the main conference. In particular, the lack of experimental demonstrations of the system, and the disconnect between domain-specific technical knowledge required to appreciate this work and that of the average ICLR attendee were some of the main causes for concern. It is clear to me that this paper is not suitable for the main conference, not due to its quality, but due to its subject matter. I would be happy, however, to tentatively recommend it for acceptance to the workshop as this topic deserves discussion at the conference, and this would provide the basis for a useful bridge between the compilers community and the deep learning community.

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR. The reviewers agree that the paper’s development of action-dependent baselines for reducing variance in policy gradient is a strong contribution and that the use of Stein's identity to provide a principled way to think about control variates is sensible. The revision clarified an number of the reviewers’ questions and the resulting paper is suitable for publication in ICLR.

(optional) Comment on this decision.


An interesting analysis of the issue of short-horizon bias in meta-optimization that highlights a real problem in a number of existing setups. I concur with Reviewer 3 that it would be nice to provide a constructive solution to this issue: if something like K-FAC does indeed work well, it would be a great addition to a final version of this paper. Nonetheless, I think the paper would be a interesting addition to ICLR and recommend acceptance.

(optional) Comment on this decision.


All the reviewers and myself have concerns about the potentially incremental nature of this work. While I do understand that the proposed method goes beyond crafting minibatch losses, and instead parametrizes things via a neural network, ultimately it's roughly very similar to simply combining MMD and minibatch discrimination and "learning  the kernel". The theoretical justifications are interesting, but the results are somewhat underwhelming (as an example, DANN's are by no means the state of the art on MNIST->MNIST_M, and this task is rather contrived; the books dataset is not even clearly used by anyone else).

The interesting analysis may make it a good candidate for the workshop track, so I am recommending that.

(optional) Comment on this decision.


  + Original regularizer that encourages discriminator representation entropy is shown to improve GAN training.
  + good supporting empirical validation
  - While intuitively reasonable, no compelling theory is given to justify the approach
  - The regularizer used in practice is a heap of heuristic approximations (continuous relaxation of a rough approximate measure of the joint entropy of a binarized activation vector)
  - The writing and the mathematical exposition could be clearer and more precise

(optional) Comment on this decision.


Reviewers largely acknowledge the novelty of the paper in proposing the use of simplex volume for measuring class scatter in few-shot learning. However there are concerns on missing comparisons with relevant baseline methods, both earlier published work as well as other simpler variants without using the volume (k-NN etc).

(optional) Comment on this decision.


The submission proposes a strategy for creating vector representations of graphs, upon which a CNN can be applied.  Although this is a useful problem to solve, there are multiple works in the existing literature for doing so.  Given that the choice between these is essentially empirical, a through comparison is necessary.  This was pointed out in the reviews, and relevant missing comparisons were given.  The authors did not provide a response to these concerns.

(optional) Comment on this decision.


Pros:
+ Use of feedforward attention for global aggregation is attractive.
+ The new kNN results for singer identification give better evidence that the learned embeddings represent singer identity.

Cons:
- The models are underfit because they are not trained on enough data.
- Problematic design of the new dataset: it is wasteful to have 30,000 songs but to be able to train on only 500 at a time.
- Narrow focus of the paper makes it more appropriate to an audio-centered venue rather than to ICLR.

The reviewers were grateful to the authors for their thorough responses in the OpenReview system and careful revision of the paper, but still feel it is not quite up to ICLR standards due to the narrow focus of the paper and the use of models that are undertrained.

(optional) Comment on this decision.


The paper studies a defense against adversarial examples that re-trains convolutional networks on adversarial examples constructed to attack pre-trained networks. Whilst the proposed approach is not very original, the paper does present a solid empirical baseline for these kinds of defenses. In particular, it goes beyond the "toy" experiments that most other studies in this space perform by experimenting on ImageNet. This is important as there is evidence suggesting that defenses against adversarial examples that work well on MNIST/CIFAR do not necessarily transfer well to ImageNet. The importance of the baseline method studied in this paper is underlined by its frequent application in the recent NIPS competition on adversarial examples.

(optional) Comment on this decision.


the problem is interesting, and the reviewers acknowledge it's worth an effort to tackle. unfortunately all the reviewers found the work to be too preliminary without a convincing evidence supporting the proposed approach against other alternatives (or on its own.)

(optional) Comment on this decision.


Two of the reviewers liked the intent of the paper -- to analyze gradient flow in residual networks and understand the tradeoffs between width and depth in such networks.  However, all reviewers flagged a number of problems in the paper, and the authors did not participate in the discussion period.

Pros:
+ Interesting analysis suggests wider, shallower ResNets should outperform narrower, deeper ResNets, and empirical results support the analysis.

Cons:
- Independence assumption on weights is not valid after any weight updates.
- The notation is not as clear as it should be.
- Empirical results would be more convincing if obtained on several tasks.
- The architecture analyzed in the paper is not standard, so it isn't clear how relevant it is for other practitioners.
- Analysis and paper should take into account other work in this area, e.g. Veit et al., 2016 and Schoenholz et al., 2017.

(optional) Comment on this decision.


The paper proposes a novel method for conditional image generation which is based on nearest neighbor matching for transferring high-frequency statistics. The evaluation is carried out on several image synthesis tasks, where the technique is shown to perform better than an adversarial baseline.

(optional) Comment on this decision.


This paper presents a new method for approximate Bayesian inference in neural networks.  The reviewers all found the proposed idea interesting but originally had questions about its novelty (with regard to normalizing flows) and questioned the technical rigor of the approach.  The authors did a good job of addressing the technical concerns, causing two of the reviewers to raise their scores.  However, the paper remains just borderline and none of the reviewers are willing to champion the paper as their questions about novelty and empirical evaluation remain.  The reviewers all questioned fundamental technical aspects of the paper (which were clarified in the discussion), indicating that the paper requires more careful exposition of the technical contributions.  Taking the reviewers feedback and discussion into account, running some more compelling experiments and rewriting the paper to make the technical aspects more clear would make this a much stronger submission.

Pros:
- Provides an interesting idea for approximate Bayesian inference in deep networks
- The paper appears correct
- The approach is scalable and tractable

Cons:
- The technical writing is not rigorous
- The reviewers don't seem convinced by the empirical analysis
- Incremental over existing (but recent) work (Luizos and Welling)

(optional) Comment on this decision.


This paper uses known methods for learning a differentially private models and applies it to the task of learning a language model, and find they are able to maintain accuracy results on large datasets. Reviewers found the method convincing and original saying it was "interesting and very important to the machine learning ... community", and that in terms of results it was a "very strong empirical paper, with experiments comparable to industrial scale". There were some complaints as to the clarity of the work, with requests for more clear explanations of the methods used.

(optional) Comment on this decision.


The submission proposes optimization with hard-threshold activations.  This setting can lead to compressed networks, and is therefore an interesting setting if learning can be achieved feasibly.  This leads to a combinatorial optimization problem due to the non-differentiability of the non-linearity.  The submission proceeds to analyze the resulting problem and propose an algorithm for its optimization.

Results show slight improvement over a recent variant of straight-through estimation (Hinton 2012, Bengio et al. 2013), called saturated straight-through estimation (Hubara et al., 2016).  Although the improvements are somewhat modest, the submission is interesting for its framing of an important problem and improvement over a popular setting.

(optional) Comment on this decision.


After careful consideration, I think that this paper in its current form is just under the threshold for acceptance. Please note that I did take into account the comments, including the reviews and rebuttals, noting where arguments may be inconsistent or misleading.

The paper is a promising extension of RCC, albeit too incremental. Some suggestions that may help for the future:

1) Address the sensitivity remark of reviewer 2. If the hyperparameters were tuned on RCV1 instead of MNIST, would the results across the other datasets remain consistent?

2) Train RCC or RCC-DR in an end-to-end way to gauge the improvement of joint optimization over alternating, as this is one of the novel contributions.

3) Discuss how to automatically tune \lambda and \delta_1 and \delta_2. These may appear in the RCC paper, but it's unclear if the same derivations hold when going to the non-linear case (they may in fact transfer gracefully, it's just not obvious). It would also be helpful for researchers building on DCC.

(optional) Comment on this decision.


The paper shows that many of the current state-of-the-art interpretability methods are inaccurate even for linear models. Then based on their analysis of linear models they propose a technique that is thus accurate for them and also empirically provides good performance for non-linear models such as DNNs.

(optional) Comment on this decision.


This is another paper, similar in spirit to the Wasserstein GAN and Cramer GAN, which uses ideas from optimal transport theory to define a more stable GAN architecture. It combines both a primal representation (with Sinkhorn loss) with a minibatch-based energy distance between distributions.
The experiments show that the OT-GAN produces sharper samples than a regular GAN on various datasets. While more could probably be done to distinguish the model from WGANs and Cramer GANs, this paper seems like a worthwhile contribution to the GAN literature and merits publication.

(optional) Comment on this decision.


The reviewers were largely agreed that the paper presented an interesting idea and has potential but needs a better empirical evaluation.  It seems that the authors largely agree and are working to improve it.

PROS:
1. Improving the speed of program synthesis is a useful problem
2. Good treatment of related work, e.g. CEGIS

CONS:
1. The approach likely does not scale
2. The architecture is underspecified making it hard to reproduce
3. Only 1 domain for evaluation

(optional) Comment on this decision.


There is overall consensus about the paper's lack of novelty and clarity.  Reviewer 1 has detailed comments that can be used to strengthen the paper.  Reviewer 3 suggests that this paper is very close to Anandkumar et al 2012, and it is not clear where the novelty lies.  Addressing these concerns of the reviewers will make the paper more acceptable to future venues.

(optional) Comment on this decision.


This paper received divergent ratings (7, 3, 3). While there is value in thorough evaluation papers, this manuscript has significant presentation issues. As all three reviewers point out, the way it is currently written, it misrepresents the claims made by Mirowski et al 2016 and over-reaches in its findings. Unfortunately, we cannot make a decision on what the manuscript may look like in future once these issues are fixed, and must reject.

(optional) Comment on this decision.


This paper is well written, clear and demonstrates neat results (on, for example, the provided video of the agent playing Mario).  However, the reviewers unanimously agreed that the paper didn't demonstrate sufficient novelty over prior work, Pathak et al. 2017 in particular, to warrant acceptance.  The authors' clarification of the method and contributions in the discussion are appreciated.  Perhaps more clearly communicating these in the manuscript, as well as conveying a clear distinction or improvement over Pathak et al. will make this a much stronger submission.

Pros:
- Well written
- Impressive results on challenging RL game playing tasks
- Improves the understanding of existing curiosity driven exploration methods

Cons:
- Insufficient novelty over previous work

(optional) Comment on this decision.


Overall I agree with the assessment of R1 that the paper touches on many interesting issues (deep learning for time series, privacy-respecting ML, simulated-to-real-world adaptation) but does not make a strong contribution to any of these. Especially with respect to the privacy-respecting aspect, there needs to be more analysis showing that the generative procedure does not leak private information (noting R1 and R3’s comments). I appreciate the authors clarifying the focus of the work, and revising the manuscript to respond to the reviews. Overall it’s a good paper on an important topic but I think there are too many issues outstanding for accept at this point.

(optional) Comment on this decision.


The paper has some interesting ideas around auto-regressive policies and estimating their entropy for exploration. The use of autoregressive policies in RL is not particularly novel, and the estimate of entropy for such models is straightforward. Finally, the experiments focus on very simple tasks.

(optional) Comment on this decision.


I am inclined to agree with R1 that there is an extensive literature on learning architectures now, and I have seen two others as part of my area chairing. This paper does not offer comparisons to existing methods for architecture learning other than very basic ones and that reduces the strength of the paper significantly. Further the broad exploration over 17 tasks is more overwhelming, than adding to an insight into the methods.

(optional) Comment on this decision.


The concerns raised by AnonReviewer3 point out that, despite the effort of the authors to bridge the SM / ML divide, there is still some work to be done. The gulf between thermodynamic limits and finite effects is oft-cited in the author response. This seems to be a catch all. This gap needs to be addressed early. The authors might even suggest some open (empirical) questions looking for these phase transitions in finite systems in cases where they think engineering has placed us "not too close".

(optional) Comment on this decision.


This work combines words and images from Tumblr to provide more fine-grained sentiment analysis than just positive-negative. The contribution is too slight, as a straightforward combination of existing architectures applied on an emotion classification task with conclusions that aren't well motivated and are not providing any comparison to existing related work on finer emotion classification.

(optional) Comment on this decision.


The paper proposes a GAN model with adaptive convolution kernels. The proposed idea is reasonable, but the novelty is somewhat minor and the experimental results are limited. More comprehensive experiments (e.g., other evaluation metrics) will strengthen the future revision of paper. No rebuttal was submitted.

(optional) Comment on this decision.


The paper presents an empirical study into sparse connectivity patterns for DenseNets.

Whilst sparse connectivity is potentially interesting, the paper does not make a strong argument for such sparse connectivity patterns: in particular, the results on ImageNet suggest that sparse connectivity performs substantially worse than full connectivity (at the same FLOPS-level, Log-DenseNet obtains ~2.5% lower accuracy than baseline DenseNet models, and the best Log-DenseNet is ~4% worse than the best DenseNet). On CamVid, both network architectures appear to perform on par.

The paper motivates the model architecture by the high memory consumption of DenseNets but, frankly, that is a very weak motivation: DenseNets are actually very memory-efficient if implemented correctly (https://arxiv.org/pdf/1707.06990.pdf). The fact that such implementations are not well-supported by TensorFlow/PyTorch is a shortcoming of those deep-learning frameworks, not in DenseNets. (In fact, the memory management features that deep-learning frameworks have implemented to make residual networks memory-efficient (for instance, caching GPU memory allocation in PyTorch) are far more complex than the "thousand lines of C++" currently needed to implement a DenseNet correctly.) Such issues will likely be resolved relatively soon by better implementations, and are hardly a good motivation for a different network architecture.

(optional) Comment on this decision.


The reviewers all agree this is a well written and interesting paper describing a novel black box adversarial attack.   There were missing relevant references in the original submission, but these have been added.  I would suggest the authors follow the reviewer suggestions on claims of generality beyond CNN; although there may not be anything obvious stopping this method from working more generally, it hasn't been tested in this work.   Even if you keep the title you might be more careful to frame the body in the context of CNN's.

(optional) Comment on this decision.


All of the reviewers find the approach interesting, but they have reservations regarding the practical impact and empirical evaluation. The paper needs improvement both on the motivation and on the experimental results by including more baseline methods and neural architectures.

(optional) Comment on this decision.


The paper proposes a modification to Adam which is intended to ensure that the direction of weight update lies in the span of the historical gradients and to ensure that the effective learning rate does not decrease as the magnitudes of the weights increase.  The reviewers wanted a clearer justification of the changes made to Adam and a more extensive evaluation, and held to this opinion after reading the authors' rebuttal and revisions.

Pros:
+ The basic idea of treating the direction and magnitude separately in the optimization is interesting.

Cons:
- Insufficient evaluation of the new method.
- More justification and analysis needed for the modifications.  For example, are there circumstances under which they will fail?
- The modification to Adam and batch-normalized softmax idea are orthogonal to one another, making for a less coherent story.
- Proposed method does not have better generalization performance than SGD.
- Concern that constraining weight vectors to the unit sphere can harm generalization.

(optional) Comment on this decision.


This work shows interesting potential applications of known machine learning techniques to the practical problem of how to devise a retina prosthesis that is the most perceptually useful. The paper suffers from a few methodological problems pointed out by the reviewers (e.g., not using the more powerful neural network encoding in the subsequent experiments of the paper), but is still interesting and inspiring in its current state.

(optional) Comment on this decision.


The paper presents a layered image generation model  (e.g., foreground vs background) using GANs. The high-level idea is interesting, but novelty is somewhat limited. For example, layered generation with VAE/GAN has been explored in Yan et al. 2016 (VAEs) and Vondrick et al. 2016 (GANs). In addition, there are earlier works for unsupervised learning of foreground/background generative models (e.g., Le Roux et al., Sohn et al.). Another critical problem is that only qualitative results on relatively simple datasets (e.g., MNIST, SVHN, CelebA) are provided as experimental results. More quantitative evaluations and additional experiments on more challenging datasets will strengthen the paper.

* N. Le Roux, N. Heess, J. Shotton, J. Winn; Learning a generative model of images by factoring appearance and shape; Neural Computation 23(3): 593-650, 2011.
** Sohn, K., Zhou, G., Lee, C., & Lee, H. Learning and selecting features jointly with point-wise gated Boltzmann machines. ICML 2013.

(optional) Comment on this decision.


I fully agree with strong positive statements in the reviews.  All reviewers agree that the paper introduces a novel and elegant twist on standard RL, wherein one agent proposes a sequence of diverse tasks to a second agent so as to accelerate the second agent's learning models of the environment.  I also concur that the empirical testing of this method is quite good.  There are strong and/or promising results in five different domains (Hallway, LightKey, MountainCar, Swimmer Gather and TrainingMarines in StartCraft). This paper would make for a strong poster at ICLR.

(optional) Comment on this decision.


This is an interesting application area, but the quality of the presentation and experimental work here is not sufficient for acceptance. The numerical ratings from reviewers are just not high enough to warrant acceptance.

(optional) Comment on this decision.


This paper proposes a method for having a meta deep learning model generate the weights of a main model given a proposed architecture.  This allows the authors to search over the space of architectures efficiently.  The reviewers agreed that the paper was very well composed, presents an interesting and thought provoking idea and provides compelling empirical analysis.  An exploration of the failure modes of the approach is highly appreciated.  The lowest score was also of quite low confidence, so the overall score should probably be one point higher.

Pros:
- Very well written and composed
- "Thought provoking"
- Some strong experimental results
- Analysis of weaker experimental results (failure modes)

Cons:
- Some weak results (also in pros, however)

(optional) Comment on this decision.


All three reviewers were positive about the paper, finding it to be on an interesting topic and with broad applicability. The results were compelling and thus the paper is accepted.

(optional) Comment on this decision.


Authors present a new multi-layered capsule network architecture, implemented an EM routing procedure, and introduced "Coordinate Addition".  Capsule architectures are gaining interest because of their ability to achieve equivariance of parts, and employ a new form of pooling called "routing" (as opposed to max pooling) which groups parts that make similar predictions of the whole to which they belong, rather than relying on spatial co-locality. New state-of-art performances are being achieved on focused datasets, for which the authors have continued the trend.

Pros:
- New significant improvement to state-of-art performance is obtained on smallNORB, both in comparison to CNN structure as well as the most recent previous implementation of capsule network.

Cons:
- Some concern arose regarding the writing of the paper and the ability to understand the material, which authors have made an effort to address.

Given the general consensus of the reviewers that this work should be accepted, the general applicability of the technology to multiple domains, and the potential impact that improvements to capsule networks may have on an early field, area chair recommends this work be accepted as a poster presentation.

(optional) Comment on this decision.


This paper forms a good contribution to the active area of adversarial training.  The main issue with the original submission was presentation quality and excessive length.  The revised version is much improved.  However, it still needs some work on the writing, in large part in the transferability section but also to clean up a large number of non-native formulations like missing/extra determiners and some awkward phrasing.  It should be carefully proofread by a native English speaker if possible.  Also, the citation formatting is incorrect (frequently using \citet instead of \citep).

(optional) Comment on this decision.


Reviewers were somewhat lukewarm about this paper, which seeks to present an analysis of the limitations of sequence models when it comes to understanding compositionality. Somewhat synthetic experiments show that such models generalise poorly on patterns not attested during training, even if the information required to interpret such patterns is present in the training data when combined with knowledge of the compositional structure of the language. This conclusion seems as unsurprising to me as it does to some of the reviewers, so I would be inclined to agree with the moderate enthusiasm two out of three reviewers have for the paper, and suggest that it be redirected to the workshop track.

Other criticisms found in the review have to do with the lack of any discussion on the topic of how to address these limitations, or what message to take home from these empirical observations. It would be good for the authors to consider how to evaluate their claims against "real" data, to avoid the accusation that the conclusion is trivial from the task set up.

Therefore, while well written, it is not clear that the paper is ready for the main conference. It could potentially generate interesting discussion, so I am happy for it to be invited to the workshop track, or failing that, to suggest that further work on this topic be done before the paper is accepted somewhere.

(optional) Comment on this decision.


The paper received highly diverging scores: 5 (R1) ,9 (R2), 4(R3). Both R1 and R3 complained about the comparisons to related methods. R3 suggested some kNN and GP baselines, while R1 mentioned concurrent work using deepnets for trafffic prediction.

R3 is real expert on field. R2 and R1, not so.
R2 review very positive, but vacuous.
Rebuttal seems to counter R1 and R3 well.

It's a close all but the AC is inclined to accept since it's an interesting application of (graph-based) deepnets.

(optional) Comment on this decision.


Nice language modeling paper with consistently high scores. The model structure is neat and the results are solid. Good ICLR-type paper with contributions mostly on the ML side and experiments on a (simple) NLP task.

(optional) Comment on this decision.


This work deals with the important task of capturing named entities in a goal-directed setting. The description of the work and the experiments are not ready for publication; for example, it is unclear whether the proposed method would have an advantage over existing methods such as the match type features that are only mentioned in Table 3 for establishing the baseline on the original bAbI dialogue dataset, but not even discussed in the paper.

(optional) Comment on this decision.


The reviewers agree that the proposed architecture is novel. However, there are issues in terms of the motivation. It would be helpful in future drafts to strengthen the argument about why the architecture is expected to be better than others. Most importantly, the gains at this stage are still incremental. A larger improvement from the new architecture would motivate more researchers to focus on this architecture.

(optional) Comment on this decision.


The reviewers agree that the paper is below threshold for acceptance in the main track (one with very low confidence), but they favor submitting the paper to the workshop track.

The paper considers policy gradient methods for two-player zero-sum Alternating Markov games.  They propose adversarial policy gradient (fairly obviously), wherein the critic estimates min rather than mean reward.   They also report promising empirical results in the game of Hex, with varying board sizes.  I found the paper to be well-written and easy to read, possibly due to revisions in the rebuttal discussions.

The reviewers consider the contribution to be small, mainly due to the fact that the key algorithmic insights were already published decades ago.  Reintroducing them is a service to the community, but its novelty is limited.  Other critiques mentioned that results in Hex only provide limited understanding of the algorithm's behavior in general Alternating Markov games.  The lack of comparison with modern methods like AlphaGo Zero was also mentioned as a limitation.

Bottom line: The paper provides a small but useful contribution to the community, as described above, and the committee recommends it for workshop.

(optional) Comment on this decision.


This paper proposes to pre-train a feature embedding, using Siamese networks, for use with few-shot learning for SVMs.  The idea is not very novel since there is a fairly large body of work in the general setting of pre-trained features + simple predictor.  In addition, the experimental results could be stronger -- there are stronger results in the literature (not cited), and better data sets for testing few-shot learning.

(optional) Comment on this decision.


This paper explores the training of CNNs which have reduced-precision activations. By widening layers, it shows less of an accuracy hit on ILSVRC-12 compared to other recent reduced-precision networks. R1 was extremely positive on the paper, impressed by its readability and the quality of comparison to previous approaches (noting that results with 2-bit activations and 4-bit weights matched FP baselines). This seems very significant to me. R1 also pointed out that the technique used the same hyperparameters as the original training scheme, improving reproducibility/accessibility. R1 asked about application to MobileNets, and the authors reported some early results showing that the technique also worked with smaller network/architectures designed for low-memory hardware. R2 was less positive on the paper, with the main criticism being that the overall technical contribution of the paper was limited. They also were concerned that the paper seemed to motivate based on reducing memory footprint, but the results were focused on reducing computation. R3 liked the simplicity of the idea and comprehensiveness of the results. Like R2, they thought the paper was limited novelty. In their response to R3, the authors defended the novelty of the paper. I tend to side with the authors that very few papers target quantization at no accuracy loss. Moreover, the paper targets training, which also receives much less attention in the model compression / reduced precision literature. Is the architecture really novel? No. But does the experimental work investigate an important tradeoff? Yes.

(optional) Comment on this decision.


This paper proposes a model which learns simultaneously the dynamics of sequential data, together with a static latent representation. The idea and motivation is interesting and the results are promising.

However, all reviewers agree that the presentation needs much more work to convey the messages correctly and convincingly. Moreover, the reviewers question some design choices and lack of discussion of the results. No rebuttal has been provided.

(optional) Comment on this decision.


The reviewers concerns regarding novelty and the experimental evaluation have been resolved accordingly and all recommend acceptance. I would recommend removing the term "unsupervised" in clustering, as it is redundant. Clustering is, by default, assumed to be unsupervised.

There is some interest in extending this to non-vision domains, however this is beyond the scope of the current work.

(optional) Comment on this decision.


The paper presents a practical approach to compute Wasserstein distance based image embeddings. The Euclidean distance in the embedded space approximates the true Wasserstein distance, thus reducing the high computation cost associated with the latter.

Pros:
- Reviewers agree that the proposed solution is novel, straightforward and well described.
- Experiment demonstrate the usefulness of such embeddings for data mining tasks such as fast computation of barycenters & geodesic analysis.

Cons:
- Though the empirical analysis is convincing, the paper lacks theoretical analysis of the approximation quality.

(optional) Comment on this decision.


Pros:
- The authors propose a new algorithm to train GAN based on Cramer distance arguing that this eases optimization compared to Wasserstein GAN.
-  Reviewers agree that the paper reads well and provides a good overview of the properties of divergence measures used for GAN training.

Cons:
- It is not clear how much the central arguments about scale sensitivity, sum invariance, and unbiased sample gradients of the distances hold true in practice and generalize.
- The reviewers do not agree the benefits of the new algorithm is clear from the experiments shown.
Given the pros/cons ,the committee feels the paper falls short of acceptance in its current form.

(optional) Comment on this decision.


The consensus among the reviewers is that this paper is not quite ready for publication for reasons I will summarize in more detail below. However, I think there are some things that are really nice about this approach, and worth calling out:

PROS:

1.  the idea of tackling tasks broadly all the way from perception through symbolic reasoning is an important direction.

2. It certainly would be useful to have a "plug and play" framework in which various knowledge sources or skills can be assembled behind a simple interface designed by the ML practioner to solve a given problem or class of problems.

3. Clearly finding ways to increase sample efficiency -- especially in a deep net approach -- is of great importance practically.

4. The writing  is good.

CONS:

1. The comparison to feedforward networks needs to be made fair in order to disentangle the benefit of the architecture from the benefit of pre-training the modules.

2. Using the very limited 2x2 grid was too low a bar for the reviewers.  The authors aim at a  more general, efficient architecture useful for a variety of tasks, and perhaps you didn't want to devote too much time to this particular task, but I think having a slam-dunk example of the power of the approach is really necessary to be convincing.

3. Given the similarity, I think more has to be done to show the intellectual contribution over Zaremba et al, the difference in motivation notwithstanding.  One way to do this is to really prove out the increased sample efficiency claim.

(optional) Comment on this decision.


As the reviewers said, it is unclear what the main contribution of the paper is.

(optional) Comment on this decision.


This paper resulted in significant discussion -- both between R2 and the authors, and between the AC, PCs, and other solicited experts.

The problem of language grounding (and instruction following) in virtual environments is clearly important, this work was one of the first in the recent resurgence, and the goal of understand what the agents have learned is clearly noble and important. In terms of raw recommendations, the majority reviewer recommendation is negative, but since concerns raised by R2 seemed subjective (which in principle is not a problem), out of abundance of caution, we solicited additional input. Unfortunately, we received feedback consistent with the concerns raised here:

-- The lack of generality of the behavior found. Even if we ignore the difficult question of why the agent prefers what it does, it's unclear how the conclusions here generalize much farther than the model and environment used; the manuscript does not provide any novel or transferable principals of the form "this kind of bias in the environment leads to this kind of bias in models with these properties".

-- We realize even providing that concrete a statement might be hard, but also missing are thorough comparisons to other kinds of models (e.g. non-deep, as asked by R1) to establish that this is a general phenomenon.

Ultimately, there is a sense that this is too narrow an analysis, too soon. If there was one architecture for learning embodied agents in 3d environments that was clearly successful and useful, then studying its properties might be interesting (even crucial).  But the dust in this space isn't settled. Our current agents are fairly poor, and so the impact of understanding the biases of a specific model trained in a specific environment seems fairly low.

Finally -- this not taken into consideration in making the decision -- it is not okay to list personal homepage domains (that may reveal author identity to ACs) as conflict domains; those are meant for institutional conflicts/domains.

(optional) Comment on this decision.


PROS:
1. well-written and clear
2. added extra comparison to dagger which shows success
3. SOTA results on open ai benchmark problem and comparison to relevant related work (Shi 2017)
4. practical applications
5. created new dataset to test harder aspects of the problem

CONS:
1. the algorithmic novelty is somewhat limited
2. some indication of scalability to real-world tasks is provided but it is limited

(optional) Comment on this decision.


To ensure that a VAE with a powerful autoregressive decoder does not ignore its latent variables, the authors propose adding an extra term to the ELBO, corresponding to a reconstruction with an auxiliary non-autoregressive decoder. This does indeed produce models that use latent variables and (with some tuning of the weight on the KL term) perform as well as the underlying autoregressive model alone. However, as the reviewers pointed out, the paper does not demonstrate the value of the resulting models. If the goal is learning meaningful latent representations, then the quality of the representations should be evaluated empirically. Currently it is not clear whether that the proposed approach would yield better representations than a VAE with a non-autoregressive decoder or a VAE with an autoregressive decoder trained using the "free bits" trick of Kingma et al. (2016). This is certainly an interesting idea, but without a proper evaluation it is impossible to judge its value.

(optional) Comment on this decision.