ICLR 2018 Conference Acceptance Decision

The reviewers found the paper meaningful but noted that they were not convinced by the experiments as they stand and the presentation was dense for them.

How your identity will be displayed with the above content.

The users who will be allowed to read the above content.

(optional) Comment on this decision.

Accept (Oral)
Accept (Poster)
Reject
Invite to Workshop Track


The reviewers agree that this paper provides a sensible mechanism for producing word embeddings that exploit correlating features in the data (e.g. texts written by the same author), but point to other work doing the same thing. The lack of direct comparison in the experimental section is troublesome, although it is entirely possible the authors' were not aware of related work. Unfortunately, the lack of an author response to the reviews makes it hard to see the argument in defense of this paper, and I must recommend rejection.

(optional) Comment on this decision.


This paper presents a very cool setup for multi task learning for learning fixed length representations for sentences.  Although the authors accept the fact that fixed length representations may not be suitable for complex, long pieces of text (often, sentences), such representations may be useful for several tasks.  They use a significantly large scale setup with six interesting tasks and show that learning generic representations for sentences across tasks is useful than learning in isolation.  Two out of three reviewers presented extensive critique of the paper and there's thorough back and forth between the reviewers and the authors.  The committee believes that this paper will add positive value to the conference.

(optional) Comment on this decision.


The paper proposes a way to find why a classifier misclassified a certain instance. It tries to find pertubations in the input space to identify the appropriate reasons for the misclassification. The reviewers feel that the idea is interesting, however, it is insufficiently evaluated.  Even for the datasets they do evaluate not enough examples of success are provided. In fact, for CelebA the results are far from flattering.

(optional) Comment on this decision.


This work is proposing an approach for ensuring classification fairness through models that encapsulate deferment criteria. On the positive side, the paper provides ideas which are conceptually interesting and novel. On the other hand, the reviewers find the technical contribution to be limited and, in some cases, challenge the practicality of the method (e.g. requirement for second set of training samples). After extensive post-rebuttal discussion, the consensus is that the above issues make the paper fall below the threshold for acceptance – even if the “out-of-scope” issue is not taken into account.

(optional) Comment on this decision.


This paper received borderline reviews. Initially, all reviewers raised a number of concerns (clarity, small improvements, etc). Even after some back and forth discussion, concerns remain, and it's clear that while the idea has potential, another round of reviewing is needed before a decision can be reached. This would be a major revision in a journal. Unfortunately, that is not possible in a conference setting and we must recommend rejection. We recommend the authors to use the feedback to make the manuscript stronger and submit to a future venue.

(optional) Comment on this decision.


Authors present a method for disease classification and localization in histopathology images. Standard image processing techniques are used to extract and normalize tiles of tissue, after which features are extracted from pertained networks. A 1-D convolutional filter is applied to the bag of features from the tiles (along the tile dimension, kernel filter size equal to dimensionality of feature vector). The max R and min R values are kept as input into a neural network for classification, and thresholding of these values provides localization for disease / non-disease.

Pro:
 - Potential to reduce annotation complexity of datasets while producing predictions and localization

Con:
- Results are not great. If anything, results re-affirm why strong annotations are necessary.
- Several reviewer concerns regarding novelty of proposed method. While authors have made clear the distinctions from prior art, the significance of those changes are debated.

Given the current pros/cons, the committee feels the paper is not ready for acceptance in its current form.

(optional) Comment on this decision.


The pros and cons of this paper cited by the reviewers (with a small amount of my personal opinion) can be summarized below:

Pros:
* The method itself seems to be tackling an interesting problem, which is feature matching between encoders within a generative model

Cons:
* The paper is sloppily written and symbols are not defined clearly
* The paper overclaims its contributions in the introduction, which are not supported by experimental results
* It mis-represents the task of decipherment and fails to cite relevant work
* The experimental setting is not well thought out in many places (see Reviewer 1's comments in particular)

As a result, I do not think this is up to the standards of ICLR at this time, although it may have potential in the future.

(optional) Comment on this decision.


the problem is interesting, and the approach is also interesting. however, the reviewers have found that this manuscript would benefit from more experiments, potentially involving some real data (even at least for evaluation) in addition to largely synthetic data sets used in the submission. i also agree with them and encourage authors to consider this option.

(optional) Comment on this decision.


Pros:
+ The paper introduces a non-trivial interpretation of MAML as hierarchical Bayesian learning and uses this perspective to develop a new variation of MAML that accounts for curvature information.

Cons:
- Relatively small gains over MAML on mini-Imagenet.
- No direct comparison against the state-of-the-art on mini-Imagenet.

The reviewers agree that the interpretation of MAML as a form of hierarchical Bayesian learning is novel, non-trivial, and opens up an interesting direction for future research.  The only concerns are that the empirical results on mini-Imagenet do not show a particularly large improvement over MAML, and there is no direct comparison to the state-of-the-art results on the task.  However, the value of the new perspective on meta-learning outweighs these concerns.

(optional) Comment on this decision.


Introduces an interesting set of problems, but the approach could be more concrete and the lack of comparison to alternatives is concerning.

(optional) Comment on this decision.


All of the reviewers found some aspects of the formulation and experiments interesting, but they found the paper hard to read and understand. Some of the components of the technique such as the state screening function (SSF) seem ad-hoc and heuristic without much justification. Please improve the exposition and remove the unnecessary component of the technique, or come up with better justifications.

(optional) Comment on this decision.


The paper presents Simple Recurrent Unit, which is characterised by the lack of state-to-gates connections as used in conventional recurrent architectures. This allows for efficient implementation, and leads to results competitive with the recurrent baselines, as shown on several benchmarks.

The submission lacks novelty, as the proposed method is essentially a special case of Quasi-RNN [Bradbury et al.], published at ICLR 2017. The comparison in Appendix A confirms that, as well as similar results of SRU and Quasi-RNN in Figures 4 and 5. Quasi-RNN has already been demonstrated to be amenable to efficient implementation and perform on a par with the recurrent baselines, so this submission doesn’t add much to that.

(optional) Comment on this decision.


Area chair is in agreement with reviewers: this is a good experiment that successfully applies specific machine learning techniques to the particular task. However, the authors have not discussed or studied the breadth of other possible methods that could also solve the given task ... besides those mentioned by the reviewers, U-Nets, and variants thereof, come to mind. Without these comparisons, the novelty and significance cannot be assessed.

Authors are encouraged to study similar works, and perform a comparison among multiple possible approaches, before submission to another venue.

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR. The idea is simple, but easy to implement and effective. The paper examines the performance fairly thoroughly across a number of different scenarios showing that the method consistently reduces variance. How this translates into final performance is complex of course, but faster convergence is demonstrated and the revised experiments in table 2 show that it can lead to improvements in accuracy.

(optional) Comment on this decision.


The paper proposes using a set of orthogonal bases that combine to form convolution kernels for CNNs leading to a significant reduction of memory usage. The main concerns raised by the reviewers were 1) clarity; 2) issues with writing and presentation of results; 3) some missing experiments. The authors released a revised version of the paper and a short summary of the enhancements. None of the reviewers changed scores following the author response. The reviews were detailed and came from those familiar with CNNs. I have decided to go with reviewer consensus.

(optional) Comment on this decision.


The paper proposes a method for architecture search using network morphisms, which allows for faster search without retraining candidate models. The results on CIFAR are worse than the state of the art, but reasonably competitive, and achieved using limited computation resources. It would have been interesting to see how the method would perform on large datasets (ImageNet) and/or other tasks and search spaces. I would encourage the authors to extend the paper with further experimental evaluation.

(optional) Comment on this decision.


While the reviewers agree that this is an important topic, there are numerous concerns novelty, correctness and limitations.

(optional) Comment on this decision.


The pros and cons of the paper are summarized below:

Pros:
* The proposed tweaks to the dynamic evaluation of Mikolov et al. 2010 are somewhat effective, and when added on top of already-strong baseline models improve them substantially

Cons:
* Novelty is limited. This is essentially a slightly better training scheme than the method proposed by Mikolov et al. 2010.
* The fair comparison against Mikolov et al. 2010 is only shown in Table 1, where a perplexity of 78.6 turns to a perplexity of 73.5. This is a decent gain, but the great majority of this is achieved by switching the optimizer from SGD to an adaptive method, which as of 2018 is a somewhat limited contribution. The remainder of the tables in the paper do not compare with the method of Mikolov et al.
* The paper title, abstract, and introduction do not mention previous work, and may give the false impression that this is the first paper to propose dynamic evaluation for neural sequence models, significantly overclaiming the paper's contribution and potentially misleading readers.

As a result, while I think that dynamic evaluation itself is useful, given the limited novelty of the proposed method and the lack of comparison to the real baseline (the simpler strategy of Mikolov et al.) in the majority of the experiments, I think this papers till falls short of the quality bar of ICLR.

Also, independent of this decision, a final note about perplexity as an evaluation measure to elaborate on the comments of reviewer 1. In general, perplexity is an evaluation measure that is useful for comparing language models of the same model class, but tends to not correlate well with model performance (e.g. ASR accuracy) across very different types of models. For example, see "Evaluation Metrics for Language Models" by Chen et al. 1998. The method of dynamic evaluation is similar to the cache-based language models that existed in 1998 in that it reinforces the model to choose similar vocabulary to that it's seen before. As you can see from this paper that the quality of perplexity of an evaluation measure falls when cache-based models are thrown into the mix, and one reason for this is that cache models, while helping perplexity greatly, tend to reinforce previous errors when errors do occur.

(optional) Comment on this decision.


This work presents a neuro-symbolic model that hopes to encode structural aspects of language into hidden space representations. The reviewers were very sympathetic to the motivation of the work, but felt that it was lacking in two significant areas:

- Baselines and empirical comparisons on the tasks presented. In particular the use of state-of-the-art systems and "previous work with much better results"
- Related work on the problem of incorporating structural aspects of language into neural systems, in particular work that uses tree structured neural systems.

(optional) Comment on this decision.


This is a very interesting paper that also seems a little underdeveloped. As noted by the reviewers, it would have been nice to see the idea applied to domains requiring function approximation to confirm that it can scale -- the late addition of Freeway results is nice, but Freeway is also by far the simplest exploration problem in the Atari suite. There also seems to be a confusion between methods such as UCB, which explore/exploit, and purely exploitative methods. The case gamma_E > 0 is also less than obvious. Given the theoretical leanings of the paper, I would strongly encourage the authors to focus on deriving an RMax-style bound for their approach.

(optional) Comment on this decision.


The proposed LAN provides a visualization of the selectivity of networks to its inputs. It takes a trained network as golden target and estimates an LAN to predict masks that can be applied on inputs to generate the same outputs.
But the significance of the proposed method is unclear, "what is the potential usage of the model?". Empirical justification of that would make it stronger.

(optional) Comment on this decision.


None of the reviewers are enthusiastic about the paper, primarily due to lack of proper evaluation.  The response of the authors towards this criticism is also not sufficient.  The final results are mixed which does not show very clearly that the presented associative model performs better than the sole seq2seq baseline that the authors use for comparison.  We think that addressing these immediate concerns would improve the quality of this paper.

(optional) Comment on this decision.


This work introduces a trainable signal representation for spherical signals (functions defined in the sphere) which are rotationally equivariant by design, by extending CNNs to the corresponding group SO(3). The method is implemented efficiently using fast Fourier transforms on the sphere and illustrated with compelling tasks such as 3d shape recognition and molecular energy prediction.

Reviewers agreed this is a solid, well-written paper, which demonstrates the usefulness of group invariance/equivariance beyond the standard Euclidean translation group in real-world scenarios. It will be a great addition to the conference.

(optional) Comment on this decision.


This paper proposes an approach for predicting transcription factor (TF) binding sites and TF-TF interaction.  The approach is interesting and may ultimately be valuable for the intended application.   But in its current state, the paper has insufficient technical novelty (e.g. relative to matching networks of Vinyals 2016), insufficient comparisons with prior work, and unclear benefit of the approach.  The reviewers also had some concerns about clarity.

(optional) Comment on this decision.


The paper presents a technique for feature map compression at inference time. As noted by reviewers, the main concern is that the method is applied to one NN architecture (SqueezeNet), which severely limits its impact and applicability to better performing state-of-the-art models.

(optional) Comment on this decision.


This work presents a strong baseline model for several NLP-ish tasks such as document classification, sentence classification, representation learning based NLI, and text matching. In terms of originality, reviewers found that "there is not much contribution in terms of technical novelty" but that "one might also conclude that we need more challenging dataset". There was significant discussion about whether it "sheds new lights on limitations of existing methods" or whether the results were "marginally surprising". In terms of quality, reviewers found it to be an "insightful analysis" and noted that these "SWEMs should be considered a strong baseline in future work".

There was significant discussion with the AC about the signficance of the work. In the opinion of the AC reviewers did were too quick to accept the authors novelty claims, and did not push them enough to include other baselines in their tables that were not overly deep model. In particular the AC felt that important numbers were left out of the experiment tables, for document classification that muddied the results. The response of the authors was:

"Moreover, fasttext and our SWEM variants all belong to the category of simpler methods (with parameter-free compositional functions). Since our motivation is to explore the necessity of employing complicated compositional functions for various NLP tasks, we do not think it is necessary for us to make any comparisons between fasttext and SWEM."

In addition when a reviewer pointed out the lack of inclusion of FOFE embeddings, the authors noted something similar

"Besides, we totally agree that developing sentence embeddings that are both simple and efficient is a very promising research direction (FOFE is a great work along this line)."

The reviewer correctly pointed out related work that shows a model very similar to what the author's propose. In general this seems like evidence that the techniques are known, not that they are significant and novel.

(optional) Comment on this decision.


The authors propose the use of Gaussian processes as the prior over activation functions in deep neural networks.  This is a purely mathematical paper in which the authors derive an efficient and scalable approach to their problem.  The idea of having flexible distributions over activation functions is interesting and possibly impactful.  One reviewer recommended acceptance with low confidence.  The other two found the idea interesting and compelling but confidently recommended rejection.  These reviewers are concerned that the paper is unnecessarily complex in terms of the mathematical exposition and that it repeats existing derivations without citation.  It is very important that the authors acknowledge existing literature for mathematical derivations.  Furthermore, the reviewers question the correctness of some of the statements (e.g. is the variational bound preserved?).  These reviewers agreed that the paper is incomplete without any empirical validation.

Pros:
- A compelling and promising idea
- The approach seems to be scalable and highly plausible

Cons:
- No experiments
- Significant issues with citing of related work
- Significant questions about the novelty of the mathematical work

(optional) Comment on this decision.


This paper has been withdrawn by the authors.

(optional) Comment on this decision.


I concur with two of the reviewers: the work is somewhat incremental in terms of technical novelty (it's effectively CycleGANs for domain adaptation with a couple of effective tricks) and the need/advantage of the cycle consistency loss is not demonstrated sufficiently. The only solid ablation evidence seems to the the SVHN-->MNIST experiment from post-submission; I would personally like to see this kind of empirical proof extended much further (the fact that Shrivastava et al.'s method doesn't work well on GTA-->Cityscapes is not itself proof that cycle consistency is needed). With more empirical evidence I can see this paper being a good candidate for a computer vision conference like CVPR or ICCV.

(optional) Comment on this decision.


The reviewers agree that the work is high quality, clear, original, and could be significant.

Despite this, the scores are borderline. The reason is due to rough agreement that the empirical evaluations are not quite there yet. In particular, two reviewers agree that, in the synthetic experiments, the method is evaluated on data that is an order of magnitude too easy and quite far from the nature of real data, which has a much lower signal to noise ratio.

However, the authors have addressed the majority of the concerns and there is little doubt that the authors are capable of carrying out this new experiment and reporting its results. Even if the results are surprising, they should shed light on what seems to be an interesting new approach.

(optional) Comment on this decision.


All reviewers believed that the novelty of the contribution was limited.

(optional) Comment on this decision.


The paper creates a dataset for exploration of RL for molecular design and I think this makes it a strong contribution to the community at the intersection of the two. For a methods focussed conference such as ICLR however, it may not be the best fit. Hence I would recommend submitting to a workshop track or targeting a more focussed venue such as a bioinformatics conference.

(optional) Comment on this decision.


This submission presents intriguingly good results on k-shot learning and I agree with the authors that the results are better than the presented previous work, and that the method is simple, so I took a deeper look into the paper despite the overall negative reviews. However, I think in its current form, the paper is not suitable for publication:

- The previous work, that the authors compare to, were not really using comparable architectures: in fact, likely much worse base models with fewer parameters etc. I think any future version of this work would need to control for architecture capacity, otherwise how can we be sure where the gains come from? To me, this is a major unknown in terms of the credit assignment for the great results.
- The authors should be comparing with MAML (and follow-up work) by Finn et al. (2017)
- I don't really understand why the authors claim to have no need for validation sets. That's a very strong claim: are ALL the hyper-parameters (model architectures etc) just chosen in another, principled way? This issue would definitely need to be addressed in a follow-up work.

(optional) Comment on this decision.


The paper presents self-training scheme for GANs. The proposed idea is simple but reasonable, and the experimental results show promise for MNIST and CIFAR10. However, the novelty of the proposed method seems relatively small and experimental results lack comparison against other stronger baselines (e.g., state-of-the-art semi-supervised methods). Presentation needs to be improved. More comprehensive experiments on other datasets would also strengthen the future version of the paper.

(optional) Comment on this decision.


The paper describes a production-ready neural text-to-speech system. The algorithmic novelty is somewhat limited, as the fully-convolutional sequence model with attention is based on the previous work. The main contribution of the paper is the description of the complete system in full detail. I would encourage the authors to expand on the evaluation part of the paper, and add more ablation studies.

(optional) Comment on this decision.


Reviewers agree that the paper is well done and addresses an interesting problem, but uses fairly standard ML techniques.
The authors have responded to rebuttals with careful revisions, and improved results.

(optional) Comment on this decision.


The paper proposes to use absolute value activations, in a joint supervised + unsupervised training (classification + deep autoencoder with tied encoder/decoder weights).
Pros:
 + simple model and approach on ideas worth revisiting
Cons:
- The paper initially approached these old ideas as novel, missing much related prior work
- It doesn't convincingly breathe novel insight into them.
- Empirical methodology is not up to standards (non-standard data split, lack of strong baselines for comparison)
- Empirical validation is too limited in scope (MNIST only).

(optional) Comment on this decision.


The paper proposes a unique network architecture that can learn divide-and-conquer strategies to solve algorithmic tasks, mimicking a class of standard algorithms.  The paper is clearly written, and the experiments are diverse.  It also seems to point in the direction of a wider class of algorithm-inspired neural net architectures.

(optional) Comment on this decision.


Pros
-- Proposes an interesting margin-based training for LMs.

Cons
-- Experimental evidence is lacking, and it’s not clear if the baselines are strong enough.

Given the reviewer comments, especially regarding experimental evaluations, the AC recommends that the paper be rejected.

(optional) Comment on this decision.


This clearly written paper describes a simple extension to hard monotonic attention -- the addition of a soft attention mechanism that operates over a fixed length window of inputs that ends at the point selected by the hard attention mechanism.  Experiments on speech recognition (WSJ) and on a document summarization task demonstrate that the new attention mechanism improves significantly over the hard monotonic mechanism.  About the only "con" the reviewers noted is that the paper is a minor extension over Raffel et al., 2017, but the authors successfully argue that the strong empirical results render this simplicity a "pro."

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR. The paper presents a general approach for handling inference in probabilistic graphical models that employ deep neural networks. The framework extends Jonhson et al. (2016) and Khan & Lin (2017). The reviewers are all in agreement that the paper is suitable for publication. The paper is well written and the use of examples to illustrate the applicability of the methods brings great clarity. The experiments are not the strongest suit of the paper and, although the revision has improved this aspect, I would encourage a more comprehensive evaluation of the proposed methods. Nevertheless, this is a strong paper.

(optional) Comment on this decision.


This is a good paper with strong results via a set of simple steps for post processing off the shelf words  embeddings.  Reviewers are enthusiastic about it and the author responses are satisfactory.

(optional) Comment on this decision.


This paper proposes a method for refining distributional semantic representation at the lexical level. The reviews are fairly unanimous in that they found both the initial version of the paper, which was deemed quite rushed, and the substantial revision unworthy of publication in their current state. The weakness of both the motivation and the experimental results, as well as the lack of a clear hypothesis being tested, or of an explanation as to why the proposed method should work, indicates that this work needs revision and further evaluation beyond what is possible for this conference. I unfortunately must recommend rejection.

(optional) Comment on this decision.


The paper presents AdvGAN: a GAN that is trained to generate adversarial examples against a convolutional network. The motivation for this method is unclear: the proposed attack does not outperform simpler attack methods such as Carlini-Wagner attack. In white-box settings, a clear downside for the attacker is that it needs to re-train its GAN everytime the defender changes its convolutional network.

More importantly, the work appears preliminary. In particular, the lack of extensive quantitative experiments on ImageNet makes it difficult to compare the proposed approach to alternative attacks methods such as (I-)FGSM, DeepFool, and Carlini-Wagner. The fact that AdvGAN performs well on MNIST is nice, but MNIST should be considered for what it is: a toy dataset. If AdvGANs are, as the authors state in their rebuttal, fast and good at generating high-resolution images, then it should be straightforward to perform comprehensive experiments with AdvGANs on ImageNet (rather than focusing on a small number of images on a single target, as the authors did in their revision)?

(optional) Comment on this decision.


The updated draft has helped to address some of the issues that the reviewers had, however the reviewers believe there are still outstanding issues. With regard to the technical flaw, one reviewer has pointed out that the update changes the story of the paper by breaking the connection between the generative and discriminative model in terms of preserving or ignoring conditional dependencies.

In terms of the experiments, the paper has been improved by the reporting of standard deviation, and comparison to other works. However it is recommended that the authors compare to NAS by fixing the number of parameters and reporting the results to facilitate an apples-to-apples comparison. Another reviewer also recommends comparing to other architectures for a fixed number of neurons.

(optional) Comment on this decision.


Predicting graphs is an interesting and important direction, and there exist essentially no (effective) general-purpose techniques for this problem.  The idea of predicting nodes one by one, though not entirely surprising, is interesting and the approach makes sense. Unfortunately, I (and some of reviewers) less convinced by evaluation:

-  For example, evaluation on syntactic parsing of natural language is very weak. First of all, the used metric -- perplexity and exact match are non-standard and problematic (e.g., optimizing exact match would largely correspond to ignoring longer sentences where predicting the entire tree is unrealistic).  Also the exact match scores are very low (~30% whereas 45+ were achieve by models back in 2010).

- A reviewer had, I believe, valid concerns about comparison with GrammarVAE, which were not fully addressed.

Overall, I believe that it is interesting work, which regretfully cannot be published as a conference paper in its current form.

+ important / under-explored problem
+ a reasonable (though maybe not entirely surprising / original) approach
- issues with evaluation

(optional) Comment on this decision.


meta score: 5

This paper gives a thorough experimental comparison of convolutional vs recurrent networks for a variety of sequence modelling tasks.  The experimentation is thorough, but the main point of the paper,  that convolutional networks are unjustly ignored for sequence modelling, is overstated as there are several areas where convolutional networks are well explored.
Pros:
 clear and well-written
 thorough set of experiments
Cons
 original contribution is not strong
 it is not as radical to consider convolutional networks for sequence modeling as the authors seem to suggest

(optional) Comment on this decision.


This paper turned out to be quite difficult to call.  My take on the pros/cons is:

1. The research topic, how and why humans can massively outperform DQN, is unanimously viewed as highly interesting by all participants.

2. The authors present an original human subject study, aiming to reveal whether human outperformance is due to human knowledge priors.  The study is well conceived and well executed.  I consider the study to be a contribution by itself.

3. The study provides prima facie evidence that human priors play a role in human performance, by changing the visual display so that the priors cannot be used.

4. However, the study is not definitive, as astutely argued by AnonReviewer2.  Experiments using RL agents (with presumably no human priors) yield behavior that is similar to human behavior.  So it is possible that some factor other than human prior may account for the behavior seen in the human experiments.

5. It would indeed be better, as argued by AnonReviewer2, to use some information-theoretic measure to distinguish the normal game from the modified games.

6. The paper has been substantially improved and cleaned up from the original version.

7. AnonReviewer1 provided some thoughtful detailed discussion of how the authors may be overstating the conclusions that one can draw from the paper.

Bottom line: Given the procs and cons of the paper, the committee recommends this for workshop.

(optional) Comment on this decision.


This is a high quality paper, clearly written, highly original, and clearly significant. The paper gives a complete analysis of SGD in a two layer network where the second layer does not undergo training and the data are linearly separable.  Experimental results confirm the theoretical suggestion that the second layer can be trained provided the weights don't change sign and remain bounded. The authors address the major concerns of the reviewers (namely, whether these results are indicative given the assumptions). This line of work seems very promising.

(optional) Comment on this decision.


Pros:
+ The paper is very clearly written.
+ The proposed re-embedding approach is easily implemented and can be integrated into fancier architectures.

Cons:
- A lot of the gains reported come from lemmatization, and the gains from background knowledge become marginal when used on a stronger baseline (e.g., ESIM with full training data and full word vectors).

This paper is rather close to the decision boundary. The authors had reasonable answers for some of the reviewers' concerns, but in the end the reviewers were not completely convinced.

(optional) Comment on this decision.


The paper proposes to use the dual formulation of the Wasserstein distance to build GAN to generate samples from a discrete distribution.

Some common concerns raised by the reviewers that are not satisfactorily addressed in the rebuttal:
- The practical implementation is quite far from the theoretical formulation.
- The experiments are very limited and performed on toy datasets. They do not clearly show that the proposed approach is effective.

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR. The big-picture idea is fairly simple, although the implementation is certainly challenging requiring a deep generative model to be trained as part of the final system. The experimental validation is not sufficient to warrant publication. A comparison to a larger number of competitors e.g. [1,2] on a greater range of tasks is required.

[1] Continual Learning Through Synaptic Intelligence Friedemann Zenke BenPoole SuryaGanguli, ICML 2017
[2] Gradient Episodic Memory for Continual Learning, David Lopez-Paz and Marc’Aurelio Ranzato, NIPS 2017

(optional) Comment on this decision.


The reviewers of the paper are not very enthusiastic of the new model proposed, nor are they very happy with the experiments presented.   It is unclear from both the POS tagging and dependency parsing results where they stand with respect to state of the art methods that do not use RNNs.  We understand that the idea is to compare various RNN architectures, but it is surprising that the authors do not show any comparisons with other methods in the literature.  The idea of truncating sequences beyond a certain length is also a really strange choice.  Addressing the concerns of the reviewers will lead to a much stronger paper in the future.

(optional) Comment on this decision.


The paper aims to address the mode collapse issue in GANs by training multiple generators and forcing them to be diverse.

Reviewers agree that the proposed solution is not novel and has disadvantages such as increased parameters due to multiple generator models. The authors do not provide convincing arguments as to why the proposed approach should work well. The experiments presented also fail to demonstrate this. The results are limited to smaller MNIST and CIFAR10 datasets. Comparisons with approaches that directly address the mode collapse problem are missing.

(optional) Comment on this decision.


All reviewers recommend accepting this paper, and this AC agrees.

(optional) Comment on this decision.


The paper presents a method for forward prediction in videos. The paper insufficiently motivates the proposed method and presents very limited empirical evaluations (no ablation studies, etc.) to backup its claims. This makes it difficult for the reader to put the work into  the context of the broader research around learning from unsupervised video data; leading reviewers to complete about perceived lack of novelty and clarity.

(optional) Comment on this decision.


This paper introduces a new dataset and method for a "semantic parsing" problem of generating logical sql queries from text.  Reviews generally seemed to be very impressed by the dataset portion of the work saying "the creation of a large scale semantic parsing dataset is fantastic," but were less compelled by the modeling aspects that were introduced and by the empirical justification for the work.  In particular:

- Several reviewers pointed out that the use of RL in particularly this style felt like it was "unjustified", and that the authors should have used simpler baselines as a way of assessing the performance of the system, e.g. "There are far simpler solutions that would achieve the same result, such as optimizing the marginal likelihood or even simply including all orderings as training examples"

- The reviewers were not completely convinced that the authors' backed up their claims about the role of this dataset as a novel contribution. In particular there were questions about its structure, e.g. "dataset only covers simple queries in form of aggregate-where-select structure" and about comparisons with other smaller but similar datasets, e.g.  "how well does the proposed model work when evaluated on an existing dataset containing full SQL queries, such as ATIS"

There was an additional anonymous discussion about the work not citing previous semantic parsing datasets. The authors noted that this discussion inappropriately brought in previous private reviews. However it seems like the main reviewers issues were orthogonal to this point, and so it was not a major aspect of this decision.

(optional) Comment on this decision.


The reviewers have found that while the task of visual domain adaptation is meaningful to explore and improve, the proposed method is not sufficiently well-motivated, explained or empirically tested.

(optional) Comment on this decision.


There was quite a bit of discussion about this paper but in the end the majority felt that, though the paper is interesting, the results are too limited and more needs to be done for publication.

PROS:
1. Good comparison of state space model variations
2. Good writing (perhaps a bit dense in places)
3. Promising results, especially concerning speedup

CONS:
1. The evaluation is quite limited

(optional) Comment on this decision.


Paper reviewed by three experts who have provided detailed feedback. All three recommend rejection, and this AC sees no reason to overrule their recommendation.

(optional) Comment on this decision.


This paper does not meet the acceptance bar this year, and thus I must recommend it for rejection.

(optional) Comment on this decision.


The reviewers unanimously recommended that this paper be accepted, as it contains an important theoretical result that there are problems for which heavy-ball momentum cannot outperform SGD. The theory is backed up by solid experimental results, and the writing is clear. While the reviewers were originally concerned that the paper was missing a discussion of some related algorithms (ASVRG and ASDCA) that were handled in discussion.

(optional) Comment on this decision.


The authors propose a hierarchical VAE model with a discrete latent variable in the top-most layer for unsupervised learning of discriminative representations.  While the reported results on the two flow cytometry datasets are encouraging, they are insufficient to draw strong conclusions about the general effectiveness of the proposed architecture. Also, as two of the reviewers stated the proposed model is very similar to several VAE models in the literature. This paper seems better suited for a more applied venue than ICLR.

(optional) Comment on this decision.


In general, the reviewers and myself find this work of some interest, though potentially somewhat incremental in terms of technical novelty compared to the work for Makhzani et al. Another bothersome aspect is the question of evaluation and understanding how well the model actually does; I am not convinced that the interpolation experiments are actually giving us a lot of insights. One interesting ablation experiment (suggested privately by one of the reviewers) would be to try AAE with Wasserstein and without a learned generator -- this would disambiguate which aspects of the proposed method bring most of the benefit. As it stands, the submission is just shy of the acceptance bar, but due to its interesting results in the natural language domain, I do recommend it being presented at the workshop track.

(optional) Comment on this decision.


At least two of the reviewers found the proposed approach novel and interesting and worthy of publication at ICLR. The reviewers raised concerns regarding the paper's terminology, which may lead to some misunderstanding. I agree that upon a quick skim, a reader may think that the paper performs the crossover operation outlined at the bottom right of Figure 1. Please consider improving the figure and the caption to prevent such a misunderstanding. You can even slightly change the title to reflect the policy distillation operation rather than naive crossover. Finally, including some more complex baselines benefits the paper. I am curious whether performing policy gradient on an ensemble of 8 policies + periodic removal of the bottom half of the policies will provide similar gains.

(optional) Comment on this decision.


This paper presents a somewhat new approach to training neural nets with ternary or low-precision weights.  However the Bayesian motivation doesn't translate into an elegant and self-tuning method, and ends up seeming kind of complicated and ad-hoc.  The results also seem somewhat toy.  The paper is fairly clearly written, however.

(optional) Comment on this decision.


 + An intriguing novel regularization method: encouraging larger norms for the feature vector input to the last softmax layer of a classifier.
 + Resonably extensive experimental validation shows that it improves test accuracy to some degree.
 - While a motivation is given, the formal analysis of what is really going on remains very superficial and limited.

Technical note: Simply scaling the softmax layer's input would not change class rankings, so any positive effect of this regularizer on classification performance is due to it changing the learning dynamic in the upper layers as well. The paper could be much stronger if it did provide an analysis regarding how the global learning dynamic is affected in all layers, by the interaction between weight decay and the last layer's feature incay.

(optional) Comment on this decision.


The paper proposes a method to map input probability distributions to output probability distributions with few parameters. They show the efficacy of their method on synthetic and real stock data. After revision they seemed to have added another dataset, however, it is not carefully analyzed like the stock data. More rigorous experimentation needs to be done to justify the method.

(optional) Comment on this decision.


The reviewers generally agree that the DDRprog method is both novel and interesting, while also seeing merit in outperformance of related methods in the empirical results. However, There were a lot of complaints about the writing quality, the clarity of the exposition, and unclear motivation of some of the work.  The reviewers also noted insufficient comparisons and discussions regarding relevant prior art, including recursive NNs, Tree RNNs, IEP, etc.  While the authors have made substantial revisions to the manuscript, with several additional pages of exposition, reviewers have not raised their scores or confidence in response.

(optional) Comment on this decision.


This paper characterizes the induced geometry of the latent space of deep generative models. The motivation is established well, such that the paper convincingly discusses the usefulness derived from these insights. For example, the results uncover issues with the currently used methods for variance estimation in deep generative models. The technique invoked to mitigate this issue does feel somehow ad-hoc, but at least it is well motivated.

One of the reviewers correctly pointed out that there is limited novelty in the theoretical/methodological aspect. However, I agree with the authors’ rebuttal in that characterizing geometries on stochastic manifolds is much less studied and demonstrated, especially in the deep learning community. Therefore, I believe that this paper will be found useful by readers of the ICLR community, and will stimulate future research.

(optional) Comment on this decision.


Paper explore depth-wise separable convolutions for sequence to sequence models with convolutions encoders.
R1 and R3 liked the paper and the results. R3 thought the presentation of the convolutional space was nice, but the experiments were hurried. Other reviewers thought the paper as a whole had dense parts and need cleaning up, but the authors seem to have only done this partially.
From the reviewers comments, I'm giving this a borderline accept. I would have been feeling much more comfortable with the decision if the authors had incorporated the reviewers' suggestions more thoroughly..

(optional) Comment on this decision.


The paper is a well-written review of regularization approaches in deep learning.
It does not offer novel approaches or novel insight with empirically demonstrated usefulness
=> ICLR is not the appropriate venue for it.

(optional) Comment on this decision.


The paper proposes a method to generate adversaries close to the (training) data manifold using GANs rather than arbitrary adversaries. They show the effectiveness of their method in terms of human evaluation and success in fooling a deep network. The reviewers feel that this paper is for the most part well-written and the contribution just about makes the mark.

(optional) Comment on this decision.


All 3 reviewers consider the paper insufficiently good, including a post-rebuttal updated score.
All reviewers + anonymous comment find that the paper isn't well-enough situated with the appropriate literature.
Two reviewers cite poor presentation - spelling /grammar errors making hte paper hard to read.
Authors have revised the paper and promise further revisions for final version.

(optional) Comment on this decision.


The idea of universal perturbation is definitely interesting and well carried out in that paper.

(optional) Comment on this decision.


The manuscript proposes a simple technique for adaptive ensemble prediction. Unfortunately, several significant concerns were raised (by R2 and R3) that this AC agrees with. Both R2 and R3 asked fairly specific questions and requested follow-up experiments, which have not been addressed.

(optional) Comment on this decision.


This paper attracted strong praise from the reviewers, who felt that it was of high quality and originality.  The broad problem that is being tackled is clearly of great importance.

This paper also attracted the attention of outside experts, who were more skeptical of the claims made by the paper. The technical merits do not seem to be in question, but rather, their interpretation/application. The perception by a community as to whether an important problem has been essentially solved can affect the choices made by other reviewers when they decide what work to pursue themselves, evaluate grants, etc. It's important that claims be conservative and highlight the ways in which the present work does not fully address the broader problem of adversarial examples.

Ultimately, it has been decided that the paper will be of great interest to the community. The authors have also been entrusted with the responsibility to consider the issues raised by the outside expert (and then echoed by the AC) in their final revisions.

One final note: In their responses to the outside expert, the authors several times remark that the guarantees made in the paper are, in form, no different from standard learning-theoretic claims: "This criticism, however, applies to many learning-theoretic results (including those applied in deep learning)." I don't find any comfort in this statement. Learning theorists have often focused on the form of the bounds (sqrt(m) dependence and, say, independence from the # of weights) and then they resort to empirical observations of correlation to demonstrate that the value of the bound is predictive for generalization. because the bounds are often meaningless ("vacuous") when evaluated on real data sets. (There are some recent examples bucking this trend.) In a sense, learning theorists have gotten off easy. Adversarial examples, however, concern security, and so there is more at stake. The slack we might afford learning theorists is not appropriate in this new context. I would encourage the authors to clearly explain any remaining work that needs to be done to move from "good enough for learning theory" to "good enough for security". The authors promise to outline important future work / open problems for the community. I definitely encourage this.

(optional) Comment on this decision.


The reviewers are concerned that the evaluation quality is not sufficient to convince readers that the proposed embedding method is indeed superior to alternatives. Though the authors attempted to address these comments in a subsequent revision but still, e.g., the evaluation is only intrinsic or on contrived problems. Given the limited novelty of the approach (it is a fairly straightforward generalization of Levy and Goldberg's factorization of PPMI matrix; the factorization is not new per se as well), the quality of experiments and analysis should be improved.

+ the paper is well written
- novelty is moderate
- better evaluation and analysis are necessary

(optional) Comment on this decision.


Meta score: 4

The paper presents a manually-constructed cloze-style fill-in-the-missing-word dataset, with baseline language modelling experiments that aim to show that  this dataset is difficult for machines relative to human performance.  The dataset is interesting but the fact that the experiments are confined to baseline language models
Pros:
 - interesting dataset
 - clear and well-written
 - attempt to move the field forward in an important area
Cons:
 - limited experimentation
 - language modelling approaches not appropriate baseline

(optional) Comment on this decision.


The submission modifies the SPEN framework for structured prediction by adding an inference network in place of the usual combinatorial optimization based inference.  The resulting architecture has some similarity to a GAN, and significantly increases the speed of inference.

The submission provides links between two seemingly different frameworks: SPENs and GANs.  By replacing inference with a network output, the connection is made, but importantly, this massively speeds up inference and may mark an important step forward in structured prediction with deep learning.

(optional) Comment on this decision.


This paper is somewhat incremental on recent prior work in a hot area; it has some weaknesses but does move the needle somewhat on these problems.

(optional) Comment on this decision.


This paper studies the problem of transfer learning in time series forecasting. The reviewers felt that this was an interesting, important topic but felt that for an empirical paper, the results were not convincing. The authors did not provide a response. I agree with the reviewers that the paper is not ready for publication at ICLR.

(optional) Comment on this decision.


Pros:
+ Interesting and promising approach to multi-domain, multi-task learning.
+ Paper is clearly written.

Cons:
- Reads more like a technical report than a research paper: more space should be devoted to explaining the design decisions behind the model and the challenges involved, as this will help others tackle similar problems.

This paper had extensive discussion between the reviewers and authors, and between the reviewers.  In the end, the reviewers want more insight into the architectural choices made, either via ablation studies or via a series of experiments in which tasks or components are added one at a time.  The consensus is that this would give readers a lot more insight into the challenges involved in tackling multiple domains and multiple tasks in a single model and a lot more guidance on how to do it.

(optional) Comment on this decision.


In this work reviewers use structured attention as a way to induce grammatical structure in NMT models. Reviewers liked th motivation of the work and found experiments mostly well done. However reviewers found the paper a bit difficult to follow, with several commenting that distinctions made between the different sub types of attention were not clear. Mainly the reviewers were not overwhelmed by the results of the work, saying that these gains, while clearly isolated to the use of structure were not significantly large. Additionally there were some concerns about the claimed novelty of the work, particularly compared to Liu and Lapata and other use of syntax in translation, and also which aspects were new or necessary.

(optional) Comment on this decision.


 + interesting approach for a detailed analysis of the limitations of autoencoders in solving a simple toy problem
 - resulting insights somewhat trivial, not really novel, nor practically useful => lacks demonstration of a gain on non-toy task
 - regularization study too limited in scope: lacking theoretical grounding, and more exhaustive comparison of regularization schemes.

(optional) Comment on this decision.


Though the general direction is interesting and relevant to ICLR, the novelty is limited. As reviewers point out it is very similar to Le & Zuidema (2015), with few modifications (using LSTM word representations, a different type of pooling). However, it is not clear if they are necessary  as there is no direct comparison (e.g., using a different type of pooling). Overall, though the submission is generally solid,  it does not seem appropriate for ICLR.

+ solid
+ well written
- novelty limited
- relation to Le & Zuidema is underplayed

(optional) Comment on this decision.


this submission proposes an efficient parametrization of a recurrent neural net by using two transition functions (one large and one small) to reduce the amount of computation (though, without actual improvement on GPU.) the reviewers found the submission very positive.

please, do not forget to include all the result and discussion on the proposed approach's relationship to VCRNN which was presented at the same conference just a year ago.

(optional) Comment on this decision.


While the paper shows some encouraging results for scaling up SVMs using coreset methods, it has fallen short of making a fully convincing case, particularly given the amount of intense interest in this topic back in the heydey of kernel methods. When it comes to scalability, it has become the norm now to benchmark results on far larger datasets using parallelism, specialized hardware in conjunction with algorithmic speedups (e.g., using random feature methods, low-rank approximations such as Nystrom and other approaches). As such the paper is unlikely to generate much interest in the ICLR community in its current form.

(optional) Comment on this decision.


The paper proposes a semi-supervised method to make deep learning more interpretable and at the same time be accurate on small datasets. The main idea is to learn dense representations from unlabelled data and then use those for building classifiers on small datasets as well as generate visual explanations. The idea is interesting, however, as one reviewer points out the presentation is poor. For instance, Table 2 is not understandable. Given the high standards of ICLR this cannot be ignored especially given the fact that the authors had the benefit of updating the paper which is a luxury for conference submissions.

(optional) Comment on this decision.


The proposed routing networks using RL to automatically learn the optimal network architecture is very interesting. Solid experimental justification and comparisons. The authors also addressed reviewers' concerns on presentation clarity in revisions.

(optional) Comment on this decision.


I think the model itself is not very novel, as pointed by the reviewers and the analysis is not very insightful either. However, the results themselves are interesting and quite good (on the copy task and pMnist, but not so much the other datasets presented (timit etc) where it not clear that long term dependencies would lead to better results). Since the method itself is not very novel, the onus is upon the authors to make a strong case for the merits of the paper --  It would be worth exploring these architectures further to see if there are useful elements for real world tasks -- more so than is demonstrated in the paper --  for example showing it on tasks such as machine translation or language modelling tasks requiring long term propagation of information or even real speech recognition, not just basic TIMIT phone frame classification rate.

As a result, while I think the paper could make for an interesting contribution, in its present form, I have settled on recommending the paper for the workshop track.

As a side note, paper is related to paper 874 in that an attention model is used to look at the past. The difference is in how the past is connected to the current model.

(optional) Comment on this decision.


The original paper was sloppy in its use of mathematical constructs such as manifolds, made assumptions that are poorly motivated (see review #2 for details), and presented an empirical evaluation is preliminary. Based on the reviews, the authors have substantially revised the paper to try and address those issues by adding new theory, etc.

Unfortunately, it is difficult to assess whether these revisions are sufficient to address the aforementioned issues without going through a second round of "full" review. I encourage the authors to use the reviewer comments to further improve the paper, and re-submit to a different venue.

(optional) Comment on this decision.


This paper compares autoencoder and GAN-based methods for 3D point cloud representation and generation, as well as new (and welcome) metrics for quantitatively evaluating generative models.  The experiments form a good but still a bit too incomplete exploration of this topic.  More analysis is needed to calibrate the new metrics.  Qualitative analysis would be very helpful here to complement and calibrate the quantitative ones.  The writing also needs improvement for clarity and verbosity.  The author replies and revisions are very helpful, but there is still some way to go on the issues above. Overall, the committee is intersting and recommends this paper for the workshop track.

(optional) Comment on this decision.


The reviewers are in agreement, that the paper is a big hard to follow and incorrect in places, including some claims not supported by experiments.

(optional) Comment on this decision.


While the reviewers are interested in the work, they find it not good enough for ICLR.
There are a number of limitations, including experimental results and presentation / language
There is no author  response / revision.

(optional) Comment on this decision.


The presented work is a good attempt to expand the work of Li and Malik to the high-dimensional, stochastic setting. Given the reviewer comments, I think the paper would benefit from highlighting the comparatively novel aspects, and in particular doing so earlier in the paper.

It is very important, given the nature of this work, to articulate how the hyperparameters of the learned optimizers, and of the hand-engineered optimizers are chosen. It is also important to ensure that the amount of time spent on each is roughly equal in order to facilitate an apples-to-apples comparison.

The chosen architectures are still quite small compared to today's standards. It would be informative to see how the learned optimizers compare on realistic architectures, at least to see the performance gap.

Please clarify the objective being optimized, and it would be useful to report test error.

The approach is interesting, but does not yet meet the threshold required for acceptance.

(optional) Comment on this decision.


This is an interesting and well-written paper introducing two unbiased gradient estimators for optimizing expectations of black box functions. LAX can handle functions of both continuous and discrete random variables, while RELAX is specialized to functions of discrete variables and can be seen as a version of the recently introduced REBAR with its concrete-relaxation-based control variate replaced by (or augmented with) a free-form function. The experimental section of the paper is adequate but not particularly strong. If Q-prop is the most similar existing RL approach, as is state in the paper, why not include it as a baseline? It would also be good to see how RELAX performs at optimizing discrete VAEs using just the free-form control variate (instead of combining it with the REBAR control variate).

(optional) Comment on this decision.


The paper proposes a method for learning convolutional networks with dynamic input-conditioned filters. There are several prior work along this idea, but there is no comparison agaist them. Overall, experimental results are not convincing enough.

(optional) Comment on this decision.


Clearly explained, well motivated and empirically supported algorithm for training deep networks while simultaneously learning their sparse connectivity.
The approach is similar to previous work (in particular Welling et al., Bayesian Learning via Stochastic Gradient Langevin Dynamics, ICML 2011) but is novel in that it satisfies a hard constraint on the network sparsity, which could be an advantage to match neuromorphic hardware limitations.

(optional) Comment on this decision.


A clearly written paper. While the practical relevance that came up in the review remains, the analysis and discussion is important for a deeper understanding of the deeper connections between these two important areas of machine learning.

(optional) Comment on this decision.


This paper presents an interesting idea that is related to imitation learning, safe exploration,
and intrinsic motivation. However, in its current state the paper needs improvement in clarity. There are also some concerns about the number of hyperparameters involved. Finally, the experimental results are not completely convincing and should reflect existing baselines in one of the areas described above.

(optional) Comment on this decision.


Using generative adversarial networks for conditional sequence generation with improved computational cost is a promising approach, unfortunately this work falls short of convincing that it would be more useful than existing methods.

(optional) Comment on this decision.


sadly, none of the reviewers seem to have been able to fully appreciate and check the proofs.

but in the words of even the least positive reviewer:
In general, I find many of the observations in this paper interesting. However, this paper is not strong enough as a theory paper; rather, the value lies perhaps in its fresh perspective.

i think we can all gain from fresh perspectives of LSTMs and DL for NLP :)

(optional) Comment on this decision.


Dear authors,

While the reviewers appreciated your analysis, they all expressed concerns about the significance of the paper. Indeed, given the plethora of GAN variants, it would have been good to get stronger evidence about the advantages of the Dudley GAN. Even though I agree it is difficult to provide a clean comparison between generative models because of the lack of clear objectives, the LL on one dataset and images generated is limited. For instance, it would have been nice to show robustness results as this is a clear issue with GANs.

(optional) Comment on this decision.


Important problem and all reviewers recommend acceptance. I agree.

(optional) Comment on this decision.


This is a thought-provoking paper that places GANs and VAEs in a single framework and, motivated by this perspective, proposes several novel extensions to them. The reviewers made several good suggestions for improving the paper and the authors are expected to make the revisions they promised. The current title of the paper is too general and should be changed to something more directly descriptive of the contents.

(optional) Comment on this decision.


This paper is a timely application of linear algebra to propose a method for reducing catastrophic interference by training a new task in a subspace of the parameter space using conceptors. The conceptors are deployed in the backprop, making this a valuable alternative to recent continual learning methods such as EWC. The paper is clearly written and the results give a clear validation of the method. The reviewers agree as to the merits of the paper.

(optional) Comment on this decision.


Two knowledgeable and confident reviewers suggest rejection, while one not confident reviewer suggests acceptance. I agree with the confident reviewers. All reviewers also point out that the paper is confusingly written and difficult to understand.

(optional) Comment on this decision.


Dear authors,

Based on the comments and your rebuttal, I am glad to accept your paper at ICLR.

(optional) Comment on this decision.


This paper does not meet the acceptance bar this year, and thus I must recommend it for rejection.

(optional) Comment on this decision.


Paper presents a way in which linear RNNs can be computed (fprop, bprop) using parallel scan. They show big improvements in speedups and show application on really long sequences. Reviews were generally favorable.

(optional) Comment on this decision.


The committee feels that this paper presents a simple, yet effective way to adapt language models from various users in a sufficiently privacy preserving way.  Empirical results are quite strong.  Reviewer 3 says that the novelty of the paper is not great, but does not provide any references to prior work that are similar to this paper.  The meta-reviewer finds the responses to Reviewer 3 sufficient to address the concerns.

Similarly, Reviewer 2 says that the paper may not be relevant to ICLR, but the committee feels its content does belong to the conference since the topic is extremely relevant to modern language processing techniques.  In fact, the authors provide several references that show that this paper is similar in content to those submissions.

Reviewer 1's concerns are also not sufficiently strong to warrant rejection.  The responses to each criticism suffices and the meta-reviewer thinks that this paper will add value to the conference.

(optional) Comment on this decision.


The paper proposes an extension to the reverse curriculum RL approach which uses a discriminator to label states as being on a goal trajectory or off the goal trajectory. The paper is well-written, with good empirical results on a number of task domains. However, the method relies on a number of assumptions on the ability of the agent to reset itself and the environment which are unrealistic and limiting, and beg the question as to why use the given method at all if this capability is assumed to exist. Overall, the method lacks significance and quality, and the motivation is not clear enough.

(optional) Comment on this decision.


Not quite enough for an oral but a very solid poster.

(optional) Comment on this decision.


The paper reports experiments where a LSTM language model is pretrained on a large corpus of reviews, and then the produced representation is used within a classifier on a number of sentiment classification datasets.  The relative success of the method is not surprising. The novelty is very questionable, the writing quality is mixed (e.g., typos, the model is not even properly described). There are many gaps in evaluation (e.g., from the intro it seems that the main focus is showing that byte level modeling is preferable to more standard set-ups -- characters / BPE / words). However, there are (almost) no experiments supporting this claim. The same is true for the 'sentiment neuron': its effectiveness is also not properly demonstrated. In general, the results are somewhat mixed.

Pros:
-- good results on some datasets
Cons:
-- limited novelty
-- some claims are not tested / issues with evaluation
-- writing quality is not sufficient / clarity issues

Overall, the reviewers are in agreement that the paper does not meet ICLR standards.

(optional) Comment on this decision.


This work attempts to incorporate affect information from additional resources into word embeddings. This is a valuable goal, but the methods used are very similar to existing ones, and the experimental results are not quite convincing enough to make a strong enough case for accepting the paper.

(optional) Comment on this decision.


The pros and cons of this paper cited by the reviewers can be summarized below:

Pros:
* The motivation of the problem is presented well
* The architecture is simple and potentially applicable to real-world applications

Cons:
* The novel methodological contribution is limited to non-existant
* Comparison against other relevant baselines is missing, and the baseline is not strong
* The evaluation methodology does not follows standard practice in IR, and thus it is difficult to analyze and compare results
* Paper is hard to read and requires proofreading

Considering these pros and cons, my conclusion is that this paper is not up to the standards of ICLR.

(optional) Comment on this decision.


Unfortunately, it falls short of ICLR standards -- from evaluation, novelty and clarity perspectives. The method is also not discussed in all details.

(optional) Comment on this decision.


Paper proposes adding randomization steps during inference time to CNNs in order to defend against adversarial attacks.

Pros:

- Results demonstrate good performance, and the team achieve a high rank (2nd place) on a public benchmark.
- The benefit of the proposed approach is that it does not require any additional training or retraining.

Cons:

- The approach is very simple, common sense would tend to suggest that adding noise to images would make adversarial attempts more difficult. Though perhaps simplicity is a good thing.
- Update: Paper does not cite related and relevant work, which takes a similar approach of requiring no retraining, but rather changing the inference stage: https://arxiv.org/pdf/1709.05583.pdf

Grammatical Suggestions:

This paper would benefit from polishing. For example:

- Abstract: sentence 1: replace “their powerful ability” to “high accuracy”
- Abstract: sentence 3: replace “I.e., clean images…” with “For example, imperceptible perturbations added to clean images can cause convolutional neural networks to fail”
- Abstract: sentence 4: replace “utilize randomization” to “implement randomization at inference time” or something similar to make more clear that this procedure is not done during training.
- Abstract: sentence 7: replace “also enjoys” with “provides”

Main Text: Capitalize references to figures (i.e. “figure 1” to “Figure 1”).

Introduction: Paragraph 4: Again, please replace “randomization” with “randomization at inference time” or something similar to better address reviewer concerns.

(optional) Comment on this decision.


The paper presents an adversarial learning framework for reading comprehension.  Although the idea is interesting and presents an approach that ideally would make reading comprehension approaches more robust, the results are not substantially solid (see reviewer 3's comments) compared to other baselines to warrant acceptance.  Comments from reviewer 2 are also noteworthy where they mention that adversarial perturbations to a context around an answer can alter the facts in the context, thus destroying the actual information present there, and the rebuttal does not seem to satisfy the concern.  Addressing these issues will strengthen the paper for a potential future venue.

(optional) Comment on this decision.


Initially this paper received mixed reviews. After reading the author response, R1 and and R3 recommend acceptance.

R2, who recommended rejecting the paper, did not participate in discussions, did not respond to author explanations, did not respond to AC emails, and did not submit a final recommendation. This AC does not agree with the concerns raised by R2 (e.g. I don't find this model to be unprincipled).

The concerns raised by R1 and R3 were important (especially e.g. comparisons to NMS) and the authors have done a good job adding the required experiments and providing explanations.

Please update the manuscript incorporating all feedback received here, including comparisons reported to the concurrent ICLR submission on counting.

(optional) Comment on this decision.


This paper proposes a new way of learning tensors representation with ring-structured decompositions rather than through Tensor Train methods. The paper investigates the mathematical properties of this decomposition and provides synthetic experiments. There was some debate, with the reviewers, about the novelty and impact of this method, where overall the feeling was this work was too preliminary to be accepted. The idea, from my understanding, is interesting and would benefit from discussion at the workshop track, but the authors are investigated to make a stronger case for the novelty of this method in any further work and, in particular, to consider showing empirical improvement on "real" data where TT methods are currently applied.

(optional) Comment on this decision.


This paper presents a simple yet effective method for weight dropping for an LSTM that requires no modification of an RNN cell's formulation.  Experimental results shows good perplexity results on benchmarks compared to many baselines.  All reviewers agree that the paper will bring good contribution to the conference.

(optional) Comment on this decision.


Meta score: 4

The paper concerns the development of a density network for estimating uncertainty in recommender systems.  The submitted paper is not very clear and it is hard to completely understand the proposed method from the way it is presented.  This makes assessing the contribution of the paper  difficult.

Pros:
 - addresses an  interesting and important problem
 - possible novel contribution

Cons:
 - poorly written, hard to understand precisely what is done
 - difficult to compare with the state-of-the-art, not helped by disorganised literature review
 - experimentation could be improved

The paper needs more work before being ready for publication.

(optional) Comment on this decision.


This paper is generally very strong. I do find myself agreeing with the last reviewer though, that tuning hyperparameters on the test set should not be done, even if others have done it in the past. (I say this having worked on similar problems myself.) I would strongly encourage the authors to re-do their experiments with a better tuning regime.

(optional) Comment on this decision.


As expressed by most reviewers, the idea of the paper is interesting:  using summarization as an intermediate representation for an auto encoder.  In addition, a GAN is used on the generator output to encourage the output to look like summaries.  They just need unpaired summaries.  Even if the idea is interesting, from the committee's perspective, important baselines are missing in the experimental section:  why would one choose to use this method if it is not competitive with other baselines that have proposed work in this vein?  One reviewer brings up the point that the method is significantly worse than a supervised baseline.  Moreover, the authors mention the work of Miao and Blunsom, but could have used one of their experimental setups to show that at least in the semi-supervised scenario, this work empirically performs as well or better than that baseline.

(optional) Comment on this decision.


Reviewers concur that the paper and the application area are interesting but that the approaches are not sufficiently novel to justify presentation at ICLR.

(optional) Comment on this decision.


This paper applies semi-supervised GAN training to regression, specifically to an autonomous driving dataset. The reviewers were consistent - they remarked that the task/setting was interesting but the experiments were not rigorous enough to draw meaningful conclusions. The authors should take into consideration the reviewer feedback on related work, baselines, and how it could improve clarity.

(optional) Comment on this decision.


The paper proposes a GAN based approach for disentangling identity (or class information) from style. The supervision needed is the identity label for each image. Overall, the reviewers agree that the paper makes a novel contribution along the line of work on disentangling 'style' from 'content'.

(optional) Comment on this decision.


The paper proposes a method to learn and explain simultaneously. The explanations are generated as part of the learning and in some sense come for free. It also goes the other way in that the explanations also help performance in simpler settings. Reviewers found the paper easy to follow and the idea has some value, however, the related work is sparse and consequently comparison to existing state-of-the-art explanation methods is also sparse. These are nontrivial concerns which should have been addressed in the main article not hidden away in the supplement.

(optional) Comment on this decision.


Both R1 and R2 suggested that Conceptors (Jaeger, 2014) had previously explored learning transformations in the context of reservoir computing. The authors acknowledged this in their response and added a reference. The main concern raised by the reviewers was lack of novelty and weak experiments (both the MNIST and depth maps were small and artificial). The authors acknowledged that it was mainly a proof of concept type of work. R1 and R2 also rejected the claim of biological plausibility (and this was also acknowledged by the authors). Though the authors have taken great care to respond in detail to each of the reviewers, I agree with the consensus that the paper does not meet the acceptance bar.

(optional) Comment on this decision.


The submission proposes a Kuiper statistic based loss function for survival clustering.  This loss function is applied to train a deep network.  Results are presented on a Friendster dataset.

This submission received borderline/mixed reviews.  The primary concerns were: justification of the Kuiper loss, lack of details of the experimental setup, writing style.  In the end, these concerns remain.

Of particular importance is the justification and experimental validation of the Kuiper statistic.  Although it seems a reasonable choice, from the authors' response to R3: "We now also report results for Kolmogorov-Smirnov loss. Although the difference in performance between the two loss functions is not significant in the Friendster dataset, Kuiper loss has higher statistical power in distinguishing distribution tails [Tygert 2010]."  If this theoretical result from [Tygert 2010] is relevant, it should be possible to demonstrate this experimentally.  If such differences are irrelevant for the data of interest, the paper should perhaps be reframed with a better discussion of available statistics and literature (cf. Reviewer 2), and a more general presentation de-emphasizing modeling choices that may have limited practical relevance.

(optional) Comment on this decision.


Joint optimization of dimensionality reduction and temporal clusters. Results suggest performance improvement in a variety of scenarios versus a baseline of a recent state-of-art clustering method.

Pro:
- Joint optimization may be new and results suggest performance improvement when done on NASA Magnetospheric Multiscale (MMS) Mission.

Con:
- Small datasets evaluated, impact unclear
- Breadth of possible applications unclear
- Similarities exist to prior works. Significance of novelty not clear.
- Unanimous consensus among reviewers that work is not in a state to be accepted.

(optional) Comment on this decision.


Viewing language modeling as a matrix factorization problem, the authors argue that the low rank of word embeddings used by such models limits their expressivity and show that replacing the softmax in such models with a mixture of softmaxes provides an effective way of overcoming this bottleneck. This is an interesting and well-executed paper that provides potentially important insight. It would be good to at least mention prior work related to the language modeling as matrix factorization perspective (e.g. Levy & Goldberg, 2014).

(optional) Comment on this decision.


This is an interesting paper and addresses an important problem of neural networks with memory constrains. New experiments have been added that add to the paper, but the full impact of the paper is not yet realised, needing further exploration of models of current practice, wider set of experiments and analysis, and additional clarifying discussion.

(optional) Comment on this decision.


While the problem of learning word embeddings for a new domain is important, the proposed method was found to be unclearly presented and missing a number of important baselines. The reviewers found the technical contribution to be of only limited value.

(optional) Comment on this decision.


This work presents a RNN tailored to generate sketch drawings. The model has novel elements and advances specific to the considered task, and allows for free generation as well as generation with (partial) input. The results are very satisfactory. Importantly, as part of this work a large dataset of sketch drawings is released. The only negative aspect is the insufficient evaluation, as pointed out by R1 who points out the need for baselines and evaluation metrics. R1’s concerns have been acknowledged by the authors but not really addressed in the revision. Still, this is a very interesting contribution.

(optional) Comment on this decision.


This is a good paper that presents state-of-the-art results on a number of challenging NLP tasks. The idea is fairly simple and clean, I therefore expect it to get adopted in the community. It also seems to work across several tasks, which is nice. At the same time it is fairly simple (train an LSTM language model and use representations from all levels of the LSTM in the input or output layer of a supervised task of interest) and hence may be more appropriate for an NLP conference than ICLR?

It is a good paper and it will get cited even though the ML contributions are modest. Accept due to the strong empirical results.

(optional) Comment on this decision.


Learning identity-preserving transformations from unlabeled data is definitely an important and useful direction. However the paper does not have convincing experiments to establish the effectiveness of the proposed method on real datasets which is a crucial limitation in my view, given that the paper is largely based on an earlier published work by Culpepper and Olshausen (2009).

(optional) Comment on this decision.


This paper addresses the question of how to regularize when starting from a pre-trained convolutional network in the context of transfer learning.  The authors propose to regularize toward the parameters of the pre-trained model and study multiple regularizers of this type.  The experiments are thorough and convincing enough.  This regularizer has been used quite a bit for shallow models (e.g. SVMs as the authors mention, but also e.g. more general MaxEnt models).  There is at least some work on regularization toward a pre-trained model also in the context of domain adaptation with deep neural networks (e.g. for speaker adaptation in speech recognition).  The only remaining novelty is the transfer learning context.  This is not a sufficiently different setting to merit a new paper on the topic.

(optional) Comment on this decision.


For the reasons given by the reviewers and given the scores of 5-5-4 the AC recommends Rejection.

(optional) Comment on this decision.


This paper addresses the problem of learning neural graph representations, based on graph filtering techniques in the vertex domain.

Reviewers agreed on the fact that this paper has limited interest in its current form, and has serious grammatical issues. The AC thus recommends rejection at this time.

(optional) Comment on this decision.


The authors use a memory-augmented neural architecture to learn solve combinatorial optimization problems.  The reviewers consider the approach worth studying, but find the authors' experimental protocol and baselines flawed.

(optional) Comment on this decision.


Given the changes to the paper, the reviewers agree that the paper meets the bar for publication at ICLR. There are some concerns regarding the practical impact on CPUs and GPUs. I ask the authors to clearly discuss the impact on different hardware. One can argue if adaptive quantization techniques are helpful, then there is a chance that future hardware will support them. All of the experiments are conducted on toy datasets. Please consider including some experiments on Imagenet as well.

(optional) Comment on this decision.


The authors propose a technique for weight pruning that leaves block diagonal weights, instead of unstructured sparse weights, leading to faster inference.  However, the experiments demonstrating the quality of the pruned models are insufficient.   The authors also discuss connections to random matrix theory; but these connections are not worked out in detail.

(optional) Comment on this decision.


The paper performs an ablation analysis on LSTM, showing that the gating component is the most important. There is little novelty in the analysis, and in its current form, its impact is rather limited.

(optional) Comment on this decision.


Dear authors,

The reviewers appreciated your work and recognized the importance of theoretical work to understand the behaviour of deep nets. That said, the improvement over existing work (especially Montufar, 2017) is minor. This, combined with the limited attraction of such work, means that the paper will not be accepted.

I acknowledge the major modifications done but it is up to the reviewers to decide whether or not they agree to re-review a significantly updated version.

(optional) Comment on this decision.


The authors use the Cayley transform representation of an orthogonal matrix to provide a parameterization of an RNN with orthogonal weights.   The paper is clearly written and the formulation is simple and elegant.  However,  I share the concerns of reviewer 3 about the significance of another method for parameterizing orthogonal RNN, as there has not been a lot of evidence that these have been useful on real problems (and indeed, on most of the toys used show the value of orthogonal RNN, one can get good results just by orthogonal initialization, e.g. as in Henaff et. al. as cited in this work).    This work does not compare experimentally against many of the other methods, e.g. https://arxiv.org/pdf/1612.00188.pdf,  the two Jing et. al. works cited, simple projection methods (either full projections at each step or stochastic projections as in Henaff et. al.).  It does not cite or compare against the approach in https://arxiv.org/pdf/1607.04903.pdf.

(optional) Comment on this decision.


meta score: 8

The paper explores mixing 16- and 32-bit floating point arithmetic for NN training with CNN and LSTM experiments on a variety of tasks

Pros:
 - addresses an important practical problem
 - very wide range of experimentation, reported in depth

Cons:
 - one might say the novelty was minor, but the novelty comes from the extensive analysis and experiments

(optional) Comment on this decision.


This paper presents several theoretical results linking deep, wide neural networks to GPs.  It even includes illuminating experiments.

Many of the results were already developed in earlier works. However, many at ICLR may be unaware of these links, and we hope this paper will contribute to the discussion.

(optional) Comment on this decision.


The paper identifies an interesting problem in sigmoid deep nets, addressed diffferently by batchnorm, and proposes a different simple fix. It shows empirically that constraining neuron's weights to sum to zero improves training of a 100 layers sigmoid MLP.
The work is currenlty limited in its theoretical contribution, and regarding the showcased practical interest of the method compared to batchnorm (it's not appplicable to RELUs and shows positive effect on optimization but not generalization).

(optional) Comment on this decision.


This paper studies to what extent adversarial training affects the properties of adversarial examples in object classification.

Reviewers found the work going in the right direction, but agreed that it needs further evidence/focus in order to constitute a significant contribution to the ICLR community. In particular, the AC encourages authors to relate their work to the growing body of (mostly concurrent) work on robust optimization and adversarial learning. For the above reasons, the AC recommends rejection at this time.

(optional) Comment on this decision.


The authors propose an efficient LSH-based method for computing unbiased gradients for softmax layers, building on (Mussmann et al. 2017). Given the somewhat incremental nature of the method, a thorough experimental evaluation is essential to demonstrating its value. The reviewers however found the experimental section weak and expressed concerns about the choice of baselines and their surprisingly poor performance.

(optional) Comment on this decision.


The reviewers unanimously agree that this paper is worth publication at ICLR. Please address the feedback of the reviewers and discuss exactly how the potential speed up rates are computed in the appendix. I speed up rates to be different for different devices.

(optional) Comment on this decision.


Understanding global optimality conditions for deep nets even in the restricted case of linear layers is a valuable contribution. Please add clarifications to ways in which the paper goes beyond the results of Kawaguchi'16, which was the main concern expressed by the reviewers.

(optional) Comment on this decision.


This paper presents an update to the method of Franceschi 2017 to optimize regularization hyperparameters, to improve stability.  However, the theoretical story isn't so clear, and the results aren't much of an improvement.  Overall, the presentation and development of the idea needs work.

(optional) Comment on this decision.


The paper proposes a way of detecting statistical interactions in a dataset based on the weights learned by a DNN. The idea is interesting and quite useful as is showcased in the experiments. The reviewers feel that the paper is also quite well written and easy to follow.

(optional) Comment on this decision.


With scores of 7-7-6  and the justification below the AC recommends acceptance.

One of the reviewers summarizes why this is a good paper as follows:

"This paper discusses several gradient based attribution methods, which have been popular for the fast computation of saliency maps for interpreting deep neural networks. The paper provides several advances:
- This gives a more unified way of understanding, and implementing the methods.
- The paper points out situations when the methods are equivalent
- The paper analyses the methods' sensitivity to identifying single and joint regions of sensitivity
- The paper proposes a new objective function to measure joint sensitivity"

(optional) Comment on this decision.


Pros:
+ The proposed large-batch, synchronous SGD method is able to generalize at larger batch sizes than previous approaches (e.g., Goyal et al., 2017).

Cons:
- Evaluation on more than one task would make the paper more convincing.
- The addition of more hyperparameters makes the proposed algorithm less appealing.
- Some theoretical justifiction of the layer-wise rate scaling would help.
- It isn't clear that the comparison to Goyal et al., 2017 is entirely fair, because that paper also had recommendations for the implementation of batch normalization, weight decay, and a momentum correction as the learning rate is scaled up, but this submission does not address any of those.

Although the revised paper addressed many of the reviewers' concerns, they still did not feel it was quite strong enough to be accepted to ICLR.

(optional) Comment on this decision.


This paper provides a method for eliminating options in multiple-answer reading comprehension tasks, based on the contents of the text, in order to reduce the "answer space" a machine reading model must consider. While there's nothing wrong with this, conceptually, reviewers have questioned whether or not this is a particularly useful process to include in a machine reading pipeline, versus having agents that understand the text well enough to select the correct answer (which is, after all, the primary goal of machine reading). Some reviewers were uncomfortable with the choice of dataset, suggesting SQuAD might be a better alternative), and why I am not sure I agree with that recommendation, it would be good to see stronger positive results on more than one dataset. At the end of the day, it is the lack of convincing experimental results showing that this method yields substantial improvements over comparable baselines which does the most harm to this well written paper, and I must recommend rejection.

(optional) Comment on this decision.


The paper makes a good theoretical contribution by formulating the GAN training as primal-dual subgradient method for convex optimization and providing convergence proof. The authors then propose a modified objective to standard GAN training, based on this formulation, that helps address the mode collapse issue.
One weak point of the paper as pointed out by reviewers is that that the experimental results are underwhelming and the approach may not scale well to high dimensional datasets / high-resolution images. Interestingly, the proposed approach is general enough to be applied to other GAN variants that may address this issue in future. I recommend acceptance.

(optional) Comment on this decision.


RDA improves on RWA, but even so, the model is inferior to the other standard RNN models. As a result R1 and R3 question the motivation for the use of this model -- something the authors should motivate.

(optional) Comment on this decision.


This paper proposes methods for replacing parts of neural networks with tensors, the values of which are efficiently estimated through factorisation methods. The paper is well written and clear, but the two main objections from reviewers surround the novelty and evaluation of the method proposed. I am conscious that the authors have responded to reviewers on the topic of novelty, but the case could be made more strongly in the paper, perhaps by showing significant improvements over alternatives. The evaluation was considered weak by reviewers, in particular due to the lack of comparable baselines.

Interesting work, but I'm afraid on the basis of the reviews, I must recommend rejection.

(optional) Comment on this decision.


Reviewers unanimous in assessment that manuscript has merits, but does not satisfy criteria for publication.

Pros:
- Potentially novel application of neural networks to survival analysis with competing risks, where only one terminal event from one risk category may be observed.

Cons:
- Incomplete coverage of other literature.
- Architecture novelty may not be significant.
- Small performance gains (though statistically significant)

(optional) Comment on this decision.


This paper studies the approximation and integration of partial differential equations using convolutional neural networks. By constraining CNN filters to have prescribed vanishing moments, the authors interpret CNN-based temporal prediction in terms of 'pde discovery'. The method is demonstrated on simple convection-diffusion simulations.

Reviewers were mixed in assessing the quality, novelty and significance of this work. While they all acknowledged the importance of future research in this area, they raised concerns about clarity of exposition (which has been improved during the rebuttal period), as well as the novelty / motivation. The AC shares these concerns; in particular, he misses a more thorough analysis of stability (under what conditions would one use this method to estimate an actual PDE and obtain some certificate of approximation?) and discussions about pitfalls (in real situations one may not know in advance the family of differential operators involved in the physical process nor the nature of the non-linearity; does the method produce a faithful approximation? why?).

Overall, the AC thinks this is an interesting submission that is still in its preliminary stage, and therefore recommends resubmitting to the worshop track at this time.

(optional) Comment on this decision.


While one reviewer did upgrade their Rating from 6 to 7, the most negative reviewer maintains: "Overall, I find this work interesting and current results surprising. However, I find it to be a preliminary work and not yet ready for publication. The paper still lacks a conclusion / a leading hypothesis / an explanation for the shown results. I find this conclusion indispensable even for a small scientific study to be published." after the rebuttal. With scores of 7-5-4 it is just not possible for the AC to recommend acceptance.

(optional) Comment on this decision.


The reviewers agree that this paper suffers from a lack of novelty and does not make sufficient contributions to warrant acceptance.

(optional) Comment on this decision.


- The paper is overall difficult to read and would benefit from a revised presentation.
- The practical relevance of the recovery conditions and algorithmic consequences of the work is not sufficiently clear or  convincing.

(optional) Comment on this decision.