The reviewers agree that while the presented result looks interesting, it is but one result. Further, one of the reviewer finds this to be a weak comparison as well.
The novelty of the approach over the paper by Ba et. al. also is in question -- good results on multiple tasks might have made it worth exploring, but the authors did not establish this to be the case convincingly.

(optional) Comment on this decision.


ICLR 2018 Conference Acceptance Decision

Observing that in contrast to classical information bottleneck, the deep variational information bottleneck (DVIB) model is not invariant to monotonic transformations of input and output marginals, the authors show how to incorporate this invariance along with sparsity in DVIB using the copula transform. The revised version of the paper addressed some of the reviewer concerns about clarity as well as the strength of the experimental section, but the authors are encouraged to improve these aspects of the paper further.

How your identity will be displayed with the above content.

The users who will be allowed to read the above content.

(optional) Comment on this decision.

Accept (Oral)
Accept (Poster)
Reject
Invite to Workshop Track


The paper proposes to launch adversarial attacks in the latent space of VAE such that the minimal change in the latent representation leads to the decoder producing an image with class predictions altered.  Given the pros/cons the paper in its current form falls short of acceptance.

Pros:
Reviewers agree that the paper is well written and easy to follow

Cons:
- The paper lacks novelty and uses standard attacks and defense methodology.
- Reviewers find the attack scenario presented is unrealistic and hence may not useful.
- Experiments lack rigorous comparisons with baselines and it is not clear if the attack in the latent space will be stronger than the attack in the input space.

(optional) Comment on this decision.


The paper was just not well enough received to warrant acceptance.

(optional) Comment on this decision.


The paper contributes to a body of empirical work towards understanding generalization in deep learning. They do this  through a battery of experiments studying "single directions" or selectivity of small groups of neurons. The reviewers that have actively participated agree that the revision is of high quality, impact, originality, and significance. The issue of a lack of prescriptiveness was raised by one reviewer. I agree with the majority that this is not necessary, but nevertheless, the revision makes some suggestions.  I urge the authors to express the appropriate amount of uncertainty regarding any prescriptions that have not been as thoroughly vetted!

(optional) Comment on this decision.


Reviewers recognize the proposed method of hierarchical extension to ALI to be potentially novel and interesting but have expressed strong concerns on the experiments section. The paper also needs to have comparisons with relevant hierarchical generative model baselines. Not suitable for publication in its current form.

(optional) Comment on this decision.


This paper does not meet the acceptance bar this year, and thus I must recommend it for rejection.

(optional) Comment on this decision.


The paper extends the earlier work on Prototypical networks to semi-supervised setting. Reviewers largely agree that the paper is well-written. There are some concerns on the incremental nature of the paper wrt to the novelty aspect but in the light of reported empirical results which show clear improvement over earlier work and given the importance of the topic, I recommend acceptance.

(optional) Comment on this decision.


This work tackles an important problem of incremental learning and does so with extensive experimentation. As pointed out by two reviewers, the idea does seem novel and interesting, but the submission would require some rewriting before being potentially accepted at a venue like ICLR. I suggest focusing the paper more on the task-incremental learning aspects, doing the ablation studies (and other changes) as requested by the reviewers, and having a rich appendix with details (with more discussion in the paper itself).

(optional) Comment on this decision.


All reviewers acknowledge that the idea of the paper is interesting but have expressed serious concerns on empirical evaluations. The paper is not suitable for publication in its current form.

(optional) Comment on this decision.


This paper introduces an algorithm for optimization of discrete hyperparameters based on compressed sensing, and compares against standard gradient-free optimization approaches.

As the reviewers point out, the provable guarantees (as is usually the case) don't quite make it to the main results section, but are still refreshing to see in hyperparameter optimization.

The method itself is relatively simple compared to full-featured Bayesopt (spearmint), although not as widely applicable.

(optional) Comment on this decision.


The paper reports unusally rapid convergence of the ResNet-56 model on CIFAR-10 when a single cycle of a cyclic learning rate schedule is used.  The effect is analyzed from several different perspectives. However, the reviewers were not convinced because the effect is only observed for one task, so they question the significance of the result. There was significant discussion of the paper by the reviewers and area chair before this decision was reached.

Pros:
+ Paper illustrates a "super-convergence" phenomenon in which training of a ResNet-56 reaches an accuracy of 92.4% on CIFAR-10 in 10,000 iterations using a single cycle of a cyclic learning rate schedule, while a more standard piecewise-constant schedule reaches 91.2% accuracy in 80,000 iterations.
+ There was partial, independent replication of the results on other tasks reported on OpenReview.

Cons:
- In the paper, the effect is shown for only one architecture and one task.
- In the paper, the effect is shown for only a single run.
- There are no error bars to indicate which differences are significant.

(optional) Comment on this decision.


Well motivated and well received by all of the expert reviewers. The AC recommends that the paper be accepted.

(optional) Comment on this decision.


this submission demonstrates an existing loop-hole (?) in rushing out new neural language models by carefully (and expensively) running hyperparameter tuning of baseline approaches. i feel this is an important contribution, but as pointed out by some reviewers, i would have liked to see whether the conclusion stands even with a more realistic data (as pointed out by some in the field quite harshly, perplexity on PTB should not be considered seriously, and i believe the same for the other two corpora used in this submission.) that said, it's an important paper in general which will work as an alarm to the current practice in the field, and i recommend it to be accepted.

(optional) Comment on this decision.


The paper proposes to regularize via a family of structured sparsity norms on the weights of a deep network.  A proximal algorithm is employed for optimization, and results are shown on synthetic data, MNIST, and CIFAR10.

Pros: the regularization scheme is reasonably general, the optimization is principled, the presentation is reasonable, and all three reviewers recommend acceptance.

Cons: the regularization is conceptually not terribly different from other kinds of regularization proposed in the literature.  The experiments are limited to quite simple data sets.

(optional) Comment on this decision.


The reviewers are unanimous that the paper is not sufficiently clear and could be improved with better empirical results.

(optional) Comment on this decision.


the reviewers seem to agree that this submission could be much more strengthened if more investigation is done in two directions: (1) the effect of different, available resources (e.g., in the comment, the authors mentioned WikiData didn't improve, and this raises a question of what kind of properties of external resources are necessary to help) and (2) alternatives to incorporating external knowledge (e.g., as pointed out by one of the reviewers, this is certainly not the only way to do so, and external knowledge has been used by other approaches for RTE earlier. how does this specific way fare against those or other alternatives?) addressing these two points more carefully and thoroughly would make this paper much more appreciated.

(optional) Comment on this decision.


meta score: 4

The paper uses a deep autoencoder to rating prediction, with experiments on netflix.

Pros
 - Proposed dense refeeding approach appears novel
 - Good experimental results

Cons
 - limited experimentation
 - main novelty (dense refeeding) is not well linked to existing data imputation approaches
 - novel contribution is otherwise quite limited

(optional) Comment on this decision.


The paper considers learning an NMT systems while pivoting through images. The task is formulated as a referential game. From the modeling and set-up perspective it is similar to previous work in the area of emergent communication / referential games, e.g., Lazaridou et al (ICLR 17) and especially to Havrylov & Titov (NIPS 17), as similar techniques are used to handle the variable-length channel (RNN encoders / decoders + the ST Gumbel-Softmax estimator).  However, its multilingual version is interesting and the results are sufficiently convincing (e.g., comparison to Nakayama and Nishida, 17). The paper would more attractive for those interested in emergent communication than the NMT community, as the set-up (using pivoting through images) may be perceived somewhat exotic by the NMT community. Also, the model is not attention-based (unlike SoA in seq2seq / NMT), and it is not straightforward to incorporate attention (see R2 and author response).

+ an interesting framing of the weakly-supervised MT problem
+ well written
+ sufficiently convincing results
- the set-up and framework (e.g., non-attention based) is questionable from practical perspective

(optional) Comment on this decision.


The paper aims to combine Wasserstein GAN with Improved GAN framework for semi-supervised learning.

The reviewers unanimously agree that:
 - the paper lacks novelty and such approaches have been tried before.
 - the approach does not make sufficient gains over the baselines and stronger baselines are missing.
 - the paper is not well written and experimental results are not satisfactory.

(optional) Comment on this decision.


The paper received borderline-negative scores (6,5,5) with R1 and R2 having significant difficulty with the clarity of the paper. Although R3 was marginally positive, they pointed out that the experiments are "extremely weak". The AC look at the paper and agrees with R3 on this point. Therefore the paper cannot be accepted in its current form. The experiments and clarity need work before resubmission to another venue.

(optional) Comment on this decision.


The reviewers are in agreement that while the paper is interesting, both the clarity of presentation and experimental rigor could be improved. The committee feels this paper is not ready for publication at ICLR 2018 inits current form.

(optional) Comment on this decision.


The reviewers liked this paper quite a bit. The novelty seems modest and the results are limited to a fairly simple NER task, but there is nothing wrong with the paper, hence recommending acceptance.

(optional) Comment on this decision.


This work is incremental compared to previous work, solving very specific challenges, and would probably appeal to only a very limited fraction of ICLR's audience.

(optional) Comment on this decision.


This paper does not seem completely appropriate for ICLR.

(optional) Comment on this decision.


The paper proposes a novel method for training a non-autoregressive machine translation model based on a pre-trained auto-regressive model. The method is interesting and the evaluation is carried out well. It should be noted, however, that the relative complexity of the training procedure (which involves multiple stages and external supervision) might limit the practical applicability and impact of the technique.

(optional) Comment on this decision.


this submission introduces soft local reordering to the recently proposed SWAN layer [Wang et al., 2017] to make it suitable for machine translation. although only in small-scale experiments, the results are convincing.

(optional) Comment on this decision.


The paper proposes a novel predictive model (e.g., from videos), called error encoding networks, by first learning a deterministic prediction model and then learning to minimize the residual error using latent variables. The latent variables given the sample are estimated by sampling from the prior then updating via gradient descent. The proposed method shows improved performance over the baselines. However, the qualitative results are not fully convincing, possibly because of (1) the limitation of the architecture, (2) suboptimal implementation/tuning of baselines (such as GAN and cVAE).

(optional) Comment on this decision.


The paper presents an interesting extension of the SkipThought idea by modeling sentence embeddings using several document-structure related information.  Out of the various kinds of evaluations presented, the coreference results are interesting -- but, they fall short by a bit (as noted by Reviewer 2) because they don't compare with recent work by Kenton Lee et al.  In summary, the idea provides an interesting bit on building sentence embeddings, but the experimental results could have been stronger.

(optional) Comment on this decision.


State-of-the-art results on Squad (at least at time of submission) with a nice model. Authors have since applied the model to additional tasks (SNLI). Good discussion with reviewers, well written submission and all reviewers suggest acceptance.

(optional) Comment on this decision.


This paper presents a simple tweak to hyperband to allow it to be run asynchonously on a large cluster, and contains reasonably large-scale experiments.

The paper is written clearly enough, and will be of interest to anyone running large-scale ML experiments.  However, it falls below the bar by:
1) Not exploring the space of related ideas more.
2) Not providing novel insights.
3) Not attempting to compare against model-based parallel approaches.

(optional) Comment on this decision.


the reviewers were not fully convinced of the setting under which the proposed bipolar activation function was found by the authors to be preferable, and neither am i.

(optional) Comment on this decision.


This work presents some of the first results on unsupervised neural machine translation. The group of reviewers is highly knowledgeable in machine translation, and they were generally very impressed by the results and the think it warrants a whole new area of research noting "the fact that this is possible at all is remarkable.". There were some concerns with the clarity of the details presented and how it might be reproduced, but it seems like much of this was cleared up in the discussion. The reviewers generally praise the thoroughness of the method, the experimental clarity, and use of ablations. One reviewer was less impressed, and felt more comparison should be done.

(optional) Comment on this decision.


The paper proposes augmenting Neural Statistician with a meta-context variable that specifies the partitioning of the latent context into the per-dataset and per-datapoint dimensions. This idea makes a lot of sense but the reviewers found the experimental section clearly insufficient to demonstrate its effectiveness convincingly. Also introducing only the unsupervised version of the model, which looks challenging to train, but performing all the experiments with the less interesting semi-supervised version makes the paper both less compelling and harder to follow.

(optional) Comment on this decision.


Pros:
- The paper proposes to use a hierarchical structure to address reconstruction issues with ALI model.
- Obtaining multiple latent representations that individually achieve a different level of reconstructions is interesting.
- Paper is well written and the authors made a reasonable attempt to improve the paper during the rebuttal period.

Cons:
- Reviewers agree that the approach lacks novelty as similar hierarchical approaches have been proposed before.
- The main goal of the paper to achieve better reconstruction in comparison to ALI without changing the latter's objective seems narrow. More analysis is needed to demonstrate that the approach out-performs other approaches that directly tackle this problem in ALI.
- The paper does not provide strong arguments as to why hierarchy works (limited to 2 levels in the empirical analysis presented in the paper).
- Semi-supervised learning as a down-stream task is impressive but limited to MNSIT.

(optional) Comment on this decision.


Pros
-- Nice way to formulate domain adaptation in a Bayesian framework that explains why autoencoder and domain difference losses are useful.

Cons
-- Model closely follows the framework, but the overall strategy is similar to previous models (but with improved rationale).
-- Experimental section can be improved. It would interesting to explore and develop the relationship between the proposed technique and Tzeng et al.

Given the aforementioned cons, the AC is recommending that the paper be rejected.

(optional) Comment on this decision.


this submission has two results; (1) it defines what it means for the optimal representation is, although this is rather uninteresting that it simply says that if the representation from a model is going to be used based on some given metric, the cost function should directly reflect it, and (2) it shows that different choices of encoding and decoding have different implications. as with most of the reviewers, i found these to be a rather weak contribution.

(optional) Comment on this decision.


With an 8-6-6 rating all reviewers agreed that this paper is past the threshold for acceptance.

The  quality of the paper appears to have increased during the review cycle due to interactions with the reviewers. The paper addresses issues related to the quality of heterogeneous data sources. The paper does this through the framework of graph convolutional networks (GCNs). The work proposes a data quality level concept defined at each vertex in a graph based on a local variation of the vertex. The quality level is used as a regularizer constant in the objective function. Experimental work shows that this formulation is important in the context of time-series prediction.

Experiments are performed on a dataset that is less prominent in the ML and ICLR community, from two commercial weather services Weather Underground and WeatherBug; however, experiments with reasonable baseline models using a "Forecasting mean absolute error (MAE)" metric seem to be well done.

The biggest weakness of this work was a lack of comparison with some more traditional time-series modelling approaches. However, the authors added an auto-regressive model into the baselines used for comparison. Some more details on this model would help.

I tend to agree with the author's assertion that: "there is limited work in ICLR  on data quality, but it is definitely one essential hurdle for any representation learning model to work in practice. ".

For these reasons I recommend a poster.

(optional) Comment on this decision.


This paper presents an interesting idea: employ GANs in a manner that guarantees the generation of differentially private data.

The reviewers liked the motivation but identified various issues. Also, the authors themselves discovered some problems in their formulation; on behalf of the community, thanks for letting the readers know.

The discovered issues will need to be reviewed in a future submission.

(optional) Comment on this decision.


Thank you for submitting you paper to ICLR. The consensus from the reviewers is that there are some interesting theoretical contributions and some promising experimental support. However, although the paper is moving in the right direction, they believe that it is not quite ready for publication.

(optional) Comment on this decision.


The paper received scores of 5,5,5, with the reviewers agreeing the paper was marginally below the acceptance threshold. The main issue, raised by both R2 and R3 was that connection between representation learning in deep nets and coding theory was not fully justified/made.  With no reviewer advocating acceptance, it is not possible to accept the paper unfortunately.

(optional) Comment on this decision.


The paper proposes a new method for interpreting the hidden units of neural networks by employing an Indian Buffet Process. The reviewers felt that the approach was interesting, but at times hard to follow and more analysis was needed. In particular, it was difficult to glean any advantage of this method over others. The authors did not provide a response to the reviews.

(optional) Comment on this decision.


All three reviewers agreed that the paper was an interesting, giving a demonstration of what quantum computer could achieve. However, they all also felt that the topic was outside the main interests of the conference and better suited to other venues, e.g. a quantum computation workshop. The AC agrees with them. Thus unfortunately, the paper cannot be accepted.

(optional) Comment on this decision.


The reviewers agree the proposed idea is relatively incremental, and the paper itself does not do an exemplary job in other areas to make up for this.

(optional) Comment on this decision.


The paper studies a adversarial attacks and defenses against convolutional networks based on a minimax formulation of the problem. Whilst this is an interesting direction of research, the present paper seems preliminary. In particular, compared to several other independent ICLR submissions, the empirical evaluation is quite weak: it does not consider the strongest known gradient-based attack (Carlini-Wagner) as baseline and does not report results on ImageNet. The reviewers identify several issues related to Lemma 1 and to the clarity of presentation.

(optional) Comment on this decision.


Learn to complete an equation by filling the blank with a missing function or numeral, and also to evaluate an expression.  Along the way learn to determine if an identity holds (e.g. sin^2(x) + cos^2(x) = 1).  They use a TreeNN with a separate node for each expression in the grammar.

PROS:
1. They've put together a new dataset of equational expressions for learning to complete an equation by filling in the blank of a missing function (or value) and function evaluation.   They've done this in a nice way with a generator and will release it.

2. They've got two interesting ideas here and they seem to work.  First, they train the network to jointly learn to manipulate symbols and to evaluate them.  This helps ground the symbolic manipulations in the validity of their evaluations.  They do this by using a common tree net for both processes with both a symbol node and a number node.  They train on identities (sin^2(x) + cos^2(x) = 1) and also on ground expressions (+(1,2) = 3).  The second idea is to help the system learn the interpretation map for the numerals like the "2" in "cos^2(x) with the actual number 2.  They do this by including equations which relate decimals with their base 10 expansion.  For example 2.5 = 2*10^0 + 5*10^-1.  The "2.5" is (I think) treated as a number and handled by the number node in the network.  The RHS leaves are treated as symbols and handled by the symbol node of the network. This lets them learn to represent decimals using just the 10 digits in their grammar and ties the interpretation of the symbols to what is required for a correct evaluation (in terms of their model this means "aligning" the node for symbol with the node for number).

3. Results are good over what seem to us reasonable baselines

CONS:

1. The architecture isn't new and the idea of representing expression trees in a hierarchical network isn't new either.

2. The writing, to me, is a bit unclear in places and I think they still have some work to do follow the reviewers' advice in this area.

I really wrestled with this one, and I appreciate the arguments that say it's not novel enough but I feel that there is something interesting in here and if the authors do a clean-up before final submission it will be ok.

(optional) Comment on this decision.


meta score: 4

This is basically an application in which some different deep learning approaches are compared on the task of automatically identifying domain names automatically generated by malware.  The experiments are well-constructed and reported.  However, the work does not have novelty beyond the application domain, and thus is not really suitable for ICLR.

Pros
 - good set of experiments carried out on an important task
 - clearly written
Cons
 - lacks technical novelty

(optional) Comment on this decision.


Pros:
+ Clear, well-written paper that tackles an interesting problem.
+ Interesting potential connections to other approaches in the literature such as Carreira-Perpiñán and Wang, 2014 and Taylor et al., 2016.
+ Paper shows good understanding of the literature, has serious experiments, and does not overstate the results.

Cons:
- Theory only addresses gradient descent, not stochastic gradient descent.
- Because the optimization process is similar to BFGS, it would make sense to have an empirical comparison against some second-order method, even though the proposed algorithm is more like standard backpropagation.

This paper is a nice first step in an interesting direction, and belongs in ICLR if there is sufficient space.

(optional) Comment on this decision.


This is an interesting paper that provides modeling improvements over several strong baselines and presents SOTA on Squad.  One criticism of the paper is that it evaluates only on Squad, which is somewhat of an artificial task, but we think for publication purposes at ICLR, the paper has a reasonable set of components.

(optional) Comment on this decision.


The reviewers rightly point out that presented analysis is limiting and that the experimental results are not extensive enough. Moreover, several existing work that use raw waveforms have interesting analysis of what the network is trying to learn. Given these comments, the AC recommends that the paper be rejected.

(optional) Comment on this decision.


The contribution of this paper basically consists of using MLPs in the attention mechanism of end-2-end memory networks. Though it leads to some improvements on bAbI (which may not be so surprising - MLP attention has been shown preferable in certain scenarious), it does not seem to be a sufficient contribution. The motivation is also confusing - the work is not really that related to relation networks, which were specifically designed to deal with situations where *relations* between objects matter. The proposed architecture does not model relations.

+  improvement on bAbI over the baselines
-  limited novelty (MLP attention is fairly standard)
-  the presentation of the idea is confusing (if the claim is about relations -> other datasets need to be considered)

There is a consensus between reviewers.

(optional) Comment on this decision.


This paper presents a sampling inference method for learning in multi-modal demonstration scenarios. Reference to imitation learning causes some confusion with the IRL domain, where this terminology is usually encountered. Providing a real application to robot reaching, while a relatively simple task in robotics, increases the difficulty and complexity of the demonstration. That makes it impressive, but also difficult to unpick the contributions and reproduce even the first demonstration. It's understandable at a meeting on learning representations that the reviewers wanted to understand why existing methods for learning multi-modal distributions would not work, and get a better understanding of the tradeoffs and limitations of the proposed method. The CVAE comparison added to the appendix during the rebuttal period just pushed this paper over the bar. The demonstration is simplified, so much easier to reproduce, making it more feasible others will attempt to reproduce the claims made here.

(optional) Comment on this decision.


The authors introduce a new activation function which is similar in shape to ELU, but is faster to compute.   The reviewers consider this to not be a significant innovation because the amount of time spent in computing the activation function is small compared to other neural network operations.

(optional) Comment on this decision.


Authors do not respond to significant criticism - e.g. lack of a critical reference
Reviewers unanimously reject.

(optional) Comment on this decision.


This paper proposes an implicit model of graphs, trained adversarially using the Gumbel-softmax trick.  The main idea of feeding random walks to the discriminator is interesting and novel.  However,
1) The task of generating 'sibling graphs', for some sort of bootstrap analysis, isn't well-motivated.
2) The method is complicated and presumably hard to tune, with two separate early-stopping thresholds that need to be tuned
3) There is not even a mention of a large existing literature on generative models of graphs using variational autoencoders.

(optional) Comment on this decision.


This paper presents a toolbox for the exploration of layerwise-parallel deep neural networks. The reviewers were consistent in their analysis of this paper: it provided an interesting class of models which warranted further investigation, and that the toolbox would be useful to those who are interested in exploring further. However, there was a lack of convincing examples, and also some concern that Theano (no longer maintained) was the only supported backend. The authors responded to say that they had subsequently incorporated TensorFlow support, they were not able to provide any more examples due to several reasons: “time, pending IP concerns, open technical details, sufficient presentation quality, page restriction.” I agree with the consensus reached by the reviewers.

(optional) Comment on this decision.


The paper received scores either side of the borderline: 6 (R1), 5 (R2), 7 (R3). R1 and R3 felt the idea to be interesting, simple and effective. R2 raised a number of concerns which the rebuttal addressed satisfactorily. Therefore the AC feels the paper can be accepted.

(optional) Comment on this decision.


This paper combines multiple existing ideas in Bayesian optimization (continuous-fidelity, use of gradient information and knowledge gradient) to develop their proposed cfKG method.  While the methodology seems neat and effective, the reviewers (and AC) found that the presented approach was not quite novel enough in light of existing work to justify acceptance to ICLR.  Continuous fidelity Bayesian optimization is well studied and knowledge gradient + derivative information was presented at NIPS.  The combination of these things seems quite sensible but not sufficiently novel (unless the empirical results were *really* compelling).

Pros:
- The paper is clear and writing is of high quality
- Bayesian optimization is interesting to the community and compelling methods are potentially practically impactful
- Outperforms existing methods on the chosen benchmarks

Cons:
- Is an incremental combination of existing methods
- The paper claims too much

(optional) Comment on this decision.


This paper presents a GAN training algorithm motivated by online learning. The method is shown to converge to a mixed Nash equilibrium in the case of a shallow discriminator. In the initial version of the paper, reviewers had concerns about weak baselines in the experiments, but the updated version includes comparisons against a variety of modern GAN architectures which have been claimed to fix mode dropping. This seems to address the main criticism of the reviewers. Overall, this paper seems like a worthwhile addition to the GAN literature.

(optional) Comment on this decision.


This work looks at what factors can lead to the emergence of selectivity (to certain categories) in units of a neural network. While this is an intriguing area to explore, this work uses settings that are quite toy-ish, making it a very hard to see how the observations could generalize to more realistic architectures or tasks.

(optional) Comment on this decision.


This paper's idea is to augment pre-trained word embeddings on a large corpus with embeddings learned on the data of interest. This is shown to yield better results than the pre-trained word embeddings alone. This contribution is too limited to justify publication at iclr.

(optional) Comment on this decision.


This paper extends last year's paper on PATE to large-scale, real-world datasets.  The model works by training multiple "teacher" models -- one per dataset, where a dataset might be for example, one user's data -- and then distilling those models into a student model. The teachers are all trained on disjoint data. Differential privacy is guaranteed by aggregating the teacher responses with added noise.  The paper shows improved teacher consensus by adding more concentrated noise and allowing the teacher to simply not respond to a student query.  The new results beat the old results convincingly on a variety of measures.

Quality and Clarity: The reviewers and I thought the paper was well written.

Originality: In some sense this work is incremental, extending and improving the existing PATE framework.  However, the extensions and new analysis are non-trivial and the results are good.

PROS:
1. Well written though difficult in places for somebody like myself who is not involved in this area.
2. Much improved scalability to real datasets
3. Good theoretical analysis supporting the extensions.
4. Comparison to related work (with a new comparison to UCI medical datasets used in the original paper and better results)

CONS:
1. Perhaps a little dense for the non-expert

(optional) Comment on this decision.


While the use of RNNs for building session-based recommender systems is certainly an important class of applications, the main strength of the paper is to propose and benchmark practical modifications to prior RNN-based systems that lead to performance improvements. The reviewers have pointed out that the writing in the paper needs improvement,  modifications are somewhat straightforward and some expected baselines such as comparisons against state of the art matrix-factorization based methods is missing. As such the paper could benefit from a revision and resubmission elsewhere.

(optional) Comment on this decision.


This paper brings recent innovations in reinforcement learning to bear on a tremendously important application, treating sepsis.  The reviewers were all compelled by the application domain but thought that the technical innovation in the work was low.  While ICLR welcomes application papers, in this instance the reviewers felt that the technical contribution was not justified well enough.  Two of the reviewers asked for a more clear discussion of the underlying assumptions of the approach (i.e. offline policy evaluation and not missing at random).  Unfortunately, lack of significant revisions to the manuscript over the discussion period seem to have precluded changes to the reviewer scores.  Overall, this could be a strong submission to a conference that is more closely tied to the application domain.

Pros:
- Very compelling application that is well motivated
- Impressive (possibly impactful) results
- Thorough empirical comparison

Cons:
- Lack of technical innovation
- Questions about the underlying assumptions and choice of methodology

(optional) Comment on this decision.


Important problem (navigation in unseen 3D environments, Doom in this case), interesting hybrid approach (mixing neural networks and path-planning). Initially, there were concerns about evaluation (proper baselines, ambiguous environments, etc). The authors have responded with updated experiments that are convincing to the reviewers. R1 did not participate in the discussion and their review has been ignored. I am supportive of this paper.

(optional) Comment on this decision.


The reviewers are unanimous in finding the work in this paper highly novel and significant.  They have provided detailed discussions to back up this assessment.  The reviewer comments surprisingly included a critique that  "the scientific content of the work has critical conceptual flaws" (!)  However, the author rebuttal persuaded the reviewers that the concerns were largely addressed.

(optional) Comment on this decision.


The authors did a good job addressing reviewer concerns and analyzing and  testing their model on interesting datasets with convincing results.

(optional) Comment on this decision.


Pros
-- Lays out bounds for multi-domain adaptation based on earlier work on a single source-target domain pair.
-- Shows gains over choosing the best source domain for a target domain, or naively combining domains.

Cons
-- The reviewers agree that the extensions are relatively straightforward extensions to single source-target pair.
-- Hard-max doesn’t consider the partial contribution of multiple source domains, and considers the worst-case scenario.
-- Soft-max addresses some of these issues; the authors provide reasonable justification for the algorithm but it’s not clear that the specific choice of \alphas leads to the tightest bound.

The reviewers noted that the authors significantly improved the paper during the revision process. The AC feels that the presented techniques would be of interest to the community and would help lead discussions towards theoretically optimal ways to do domain adaptation given multiple domains. The authors are therefore encouraged to submit to the workshop track.

(optional) Comment on this decision.


This paper presents impressive results on scaling GANs to ILSVRC2012 dataset containing a large number of classes. To achieve this, the authors propose "spectral normalization" to normalize weights and stabilize training which turns out to help in overcoming mode collapse issues.  The presented methodology is principled and well written. The authors did a good job in addressing reviewer's comments and added more comparative results on related approaches to demonstrate the superiority of the proposed methodology. The reviewers agree that this is a great step towards improving the training of GANs.  I recommend acceptance.

(optional) Comment on this decision.


PROS:

1. Interesting and clearly useful idea
2. The paper is clearly written.
3. This work doesn't seem that original from an algorithmic point of view since Reed & De Freitas (2015) and Cai et. al (2017) among others have considered using execution traces. However the application to program repair is novel (as far as I know).
4. This work can be very useful for an educational platform though a limitation is the need for adding instrumentation print statements by hand.

CONS:

1. The paper has some clarity issues which the authors have promised to fix.

---

(optional) Comment on this decision.


This paper presents a nice approach to domain adaptation that improves empirically upon previous work, while also simplifying tuning and learning.

(optional) Comment on this decision.


The paper characterizes the latent space of adversarial examples and introduces the concept of local intrinsic dimenstionality (LID). LID  can be used to detect adversaries as well build better attacks as it characterizes the space in which DNNs might be vulnerable. The experiments strongly support their claim.

(optional) Comment on this decision.


meta score: 7

The paper introduces an online distillation technique to parallelise large scale training.  Although the basic idea is not novel, the presented experimentation indicates that the authors' have made the technique work.  Thus this paper should be of interest to practitioners.

Pros:
 - clearly written, the approach is well-explained
 - good experimentation on large-scale common crawl data with 128-256 GPUs
 - strong experimental results

Cons:
 - the idea itself is not novel
 - the range of experimentation could be wider (e.g. different numbers of GPUs) but this is expensive!

Overall the novelty is in making this approach work well in practice, and demonstrating it experimentally.

(optional) Comment on this decision.


The paper presents a variational Bayesian approach for quantising neural network weights and makes interesting and useful steps in this increasingly popular area of deep learning.

(optional) Comment on this decision.


This is a paper introducing a hierarchical RL method which incorporates the learning of a latent space, which enables the sharing of learned skills.

The reviewers unanimously rate this as a good paper. They suggest that it can be further improved by demonstrating the effectiveness through more experiments, especially since this is a rather generic framework. To some extent, the authors have addressed this concern in the rebuttal.

(optional) Comment on this decision.


Three reviewers recommended rejection and there was no rebuttal to overturn their recommendation.

(optional) Comment on this decision.


This paper proposes a multiscale variant of Graph Convolutional Networks (GCN) , obtained by combining separate GCN modules using powers of normalized adjacency as generators. The model is tested on several node classification semi-supervised tasks obtaining excellent numerical performance.

Reviewers acknowledged the good empirical performance of the model, but all raised the issue of limited novelty, relative to the growing body of literature on graph neural networks. In particular, they missed an analysis that compares random walks powers to other multiscale approaches and justifies its performance in the context of semi-supervised learning. Overall, the AC believes this is a good paper, but it can be significantly stronger with an extra iteration that addresses these limitations.

(optional) Comment on this decision.


This paper gives a scalable Laplace approximation which makes use of recently proposed Kronecker-factored approximations to the Gauss-Newton matrix. The approach seems sound and useful. While it is a rather natural extension of existing methods, it is well executed, and the ideas seem worth putting out there.

(optional) Comment on this decision.


This submission is a continuation of a line of theoretical work that seeks to characterize optimization landscapes of neural networks by the presence or absence of spurious local minima.  As the number of critical points grows combinatorially for larger networks, it is very challenging to show such results.  The present submission extends slightly previous work by considering two hidden units and their proof technique goes beyond that of Brutzkus and Globerson, 2017, potentially leading to more interesting results if they can be extended to more complex networks.

The setting of two hidden units is quite limited - far from any practical setting.  If this were the stepping stone to proving optimality of certain optimization strategies for more complex networks, this may be of some interest, but it seems doubtful.  One indication is given in Sec. 7 / Fig. 1 in which it is shown that for even quite small numbers of hidden units, spurious local optima do occur and are reached 40% of the time for random initializations even with only 11 nodes.

(optional) Comment on this decision.


The authors provide an extension to GCNs of Kipf and Welling in order to incorporate information about higher order neighborhoods. The extension is well motivated (and  though I agree that it is not trivial modification of the K&W approach to the second order,  thanks to the authors for the clarification).  The improvements are relatively moderate.

Pros:
-- The approach is well motivated
-- The paper is clearly written
Cons:
-- The originality and impact (as well as motivation) are questioned by the reviewers

(optional) Comment on this decision.


To summarize the pros and cons:

Pro:
* Interesting application
* Impressive results on a difficult task
* Nice discussion of results and informative examples
* Clear presentation, easy to read.

Con:
* The method appears to be highly specialized to the four bug types. It is not clear how generalizable it will be to more complex bugs, and to the real application scenarios where we are dealing with open world classification and there is not fixed set of possible bugs.

There were additional reviewer complaints that comparison to the simple seq-to-seq baseline may not be fair, but I believe that these have been addressed appropriately by the author's response noting that all other reasonable baselines require test cases, which is an extra data requirement that is not available in many real-world applications of interest.

This paper is somewhat on the borderline, and given the competitive nature of a top conference like ICLR I feel that it does not quite make the cut. It is definitely a good candidate for presentation at the workshop however.

(optional) Comment on this decision.


The authors propose to detect anomaly based on its representation quality in the latent space of the GAN trained on valid samples.

Reviewers agree that:
- The proposed solution lacks novelty and similar approaches have been tried before.
- The baselines presented in the paper are primitive and hence do not demonstrate the clear benefits over traditional approaches.

(optional) Comment on this decision.


An interesting application of graph neural networks to robotics. The body of a robot is represented as a graph, and the agent’s policy is defined using a graph neural network (GNNs/GCNs) over the graph structure.

The GNN-based policy network perform on par with best methods on traditional benchmarks, but shown to be very effective for transfer scenarios: changing robot size or disabling its components.  I believe that the reviewers' concern that the original experiments focused solely on centepedes and snakes were (at least partially) addressed in the author response: they showed that their GNN-based model outperforms MLPs on a dataset of 2D walkers.

Overall:
-- an interesting application
-- modeling robot morphology is an under-explored direction
-- the paper is  well written
-- experiments are sufficiently convincing (esp. after addressing the concerns re diversity and robustness).

(optional) Comment on this decision.


The submission addresses the problem of multiset prediction, which combines predicting which labels are present, and counting the number of each object.  Experiments are shown on a somewhat artificial MNIST setting, and a more realistic problem of the COCO dataset.

There were several concerns raised by the reviewers, both in terms of the clarity of presentation (Reviewer 1), and that the proposed solution is somewhat heuristic (Reviewer 3).  On the balance, two of three reviewers did not recommend acceptance.

(optional) Comment on this decision.


This paper presents a method for learning more flexible prior distributions for GANs by learning another distribution on top of the latent codes for training examples. It's reminiscent of layerwise training of deep generative models. This seems like a reasonable thing to do, but it's probably not a substantial enough contribution given that similar things have been done for various other generative models. Experiments show improvement in samples compared with a regular GAN, but don't compare against various other techniques that have been proposed for fixing mode dropping. For these reasons, as well as various issues pointed out by the reviewers, I don't recommend acceptance.

(optional) Comment on this decision.


The pros and cons of the paper under consideration can be summarized below:

Pros:
* Reviewers thought the underlying model is interesting and intuitive
* Main contributions are clear

Cons:
* There is confusion between keywords and topics, which is leading to a somewhat confused explanation and lack of clear comparison with previous work. Because of this, it is hard to tell whether the proposed approach is clearly better than the state of the art.
* Typos and grammatical errors are numerous

As the authors noted, the concerns about the small dataset are not necessarily warranted, but I would encourage the authors to measure the statistical significance of differences in results, which would help alleviate these concerns.

An additional comment: it might be worth noting the connections to query-based or aspect-based summarization, which also have a similar goal of performing generation based on specific aspects of the content.

Overall, the quality of the paper as-is seems to be somewhat below the standards of ICLR (although perhaps on the borderline), but the idea itself is novel and results are good. I am not recommending it for acceptance to the main conference, but it may be an appropriate contribution for the workshop track.

(optional) Comment on this decision.


Evaluating simple baselines for continuous control is important and nearest neighbor search methods are interesting. However, the reviewers think that the paper lacks citation and comparison to some prior work and evaluation on more challenging benchmarks.

(optional) Comment on this decision.


The paper proposes a multitask deep learning method (called Deep-AMFTL) for preventing negative transfer. Despite some positive experimental results, the contribution of the paper is not sufficient for publication at ICLR due to several issues: similarity between the proposed method and existing method (e.g., AMTL), unclear rationale/intuition of the proposed model, clarity of presentation, technical formulation, and limited empirical evaluations (see reviewer comments for details). No author rebuttal was submitted.

(optional) Comment on this decision.


The authors present a centralized neural controller for multi-agent reinforcement learning.   The reviewers are are not convinced that there is sufficient novelty, considering the authors setup as essentially a special case of other recent works, with added adjustments to the neural-networks that are standard in the literature.

I personally am more bullish about this paper than the reviewers, as I think engineering an architecture to perform well in interesting scenarios is worth reporting.  However, the reviewers are mostly in agreement, and their reviews were neither sloppy nor factually incorrect.  So I will recommend rejection, following their judgement.

Nevertheless, I encourage the authors to continue strengthening the results and the presentation and resubmit.

(optional) Comment on this decision.


there are two separate ideas embedded in this submission; (1) language modelling (with the negative sampling objective by mikolov et al.) is a good objective to use for extracting document representation, and (2) CNN is a faster alternative to RNN's, both of which have been studied in similar contexts earlier (e.g., paragraph vectors, CNN classifiers and so on, most of which were pointed out by the reviewers already.) Unfortunately reading this manuscript does not reveal too clearly how these two ideas connect to each other (and are separate from each other) and are related to earlier approaches, which were again pointed out by the reviewers. in summary, i believe this manuscript requires more work to be accepted.

(optional) Comment on this decision.


Pros:
-- Use of Bloomier filters for lossy compression of nets is novel and well motivated, with interesting compression performance.
Cons:
-- Does lossy compression for transmission, doesn’t address FLOPS required for runtime execution. A lot of times, client devices do not have enough cpu to run large networks (title should be udpated to mean compression and transmission)
-- Missing results for full network, larger deeper network.

Overall, the content is novel and interesting, so I would encourage the authors to submit to the workshop track.

(optional) Comment on this decision.


The reviewers have significantly different views, with one strongly negative,
one strongly positive, and one borderline negative.  However, all three
reviews seem to regard the NaaA framework as a very interesting and novel approach to training neural nets.  They also concur that the major issue with the paper is very confusing technical exposition regarding the motivation, math details, and how the idea works.  The authors indicate that they have significantly revised the manuscript to improve the exposition, but none of the reviewers have changed their scores.  One reviewer states that "technical details are still too heavy to easily follow."  My own take regarding the current section 3 is that it is still very challenging to parse and follow. Given this analysis, the committee recommends this for workshop.

Pros:
        Interesting and novel framework for training NNs
        "Adaptive DropConnect" algorithm contribution
        Good empirical results in image recognition and ViZDoom domains

Cons:
        Technical exposition is very challenging to parse and follow
        Some author rebuttals do not inspire confidence.  For example,
motivation of method due to "$100 billion market cap of Bitcoin" and in reply to unconvincing neuroscience motivation, saying "throw away the typical image of auction."

(optional) Comment on this decision.


Important problem (analyzing the properties of emergent languages in multi-agent reference games), a number of interesting analyses (both with symbolic and pixel inputs), reaching a finding that varying the environment and restrictions on language result in variations in the learned communication protocols (which in hindsight is that not surprising, but that's hindsight). While the pixel experiments are not done with real images, it's an interesting addition the literature nonetheless.

(optional) Comment on this decision.


Pros
-- Extends embeddings to use a richer representation; simple yet interesting improvement on Mikolov et al. work.
Cons
-- All of the reviewers pointed out that the experimental evaluations needs improvement. The authors should find better ways to improve both quantitative (e.g., accuracy in analogies as in Mikolov et al., or by using the model for an external task if that’s the end goal) and qualitative (using functional similarity for the baseline) evaluations.

Given these comments, the AC recommends that the paper be rejected.

(optional) Comment on this decision.


The idea of using wavelet pooling is novel and will bring many interesting research work in this direction. But more thorough experimental justification such as those recommended by the reviewers would make the paper better. Overall, the committee feels this paper will bring value to the conference.

(optional) Comment on this decision.


All three reviewers recommend acceptance. Good work, accept

(optional) Comment on this decision.


This paper investigates emergence of language from raw pixels in a two-agent setting. The paper received divergent reviews, 3,6,9. Two ACs discussed this paper, due to a strong opinion from both positive and negative reviewers. The ACs agree that the score "9" is too high: the notion of compositionality is used in many places in the paper (and even in the title), but never explicitly defined. Furthermore, the zero-shot evaluation is somewhat disappointing. If the grammar extracted by the authors in sec. 3.2 did indeed indicate the compositional nature of the emergent communication, the authors should have shown that they could in fact build a message themselves, give it to the listener with an image and ask it to answer. On the other hand, "3" is also too low of a score. In this renaissance of emergent communication protocol with multi-agent deep learning systems, one missing piece has been an effort toward seriously analyzing the actual properties of the emergent communication protocol.  This is one of the few papers that have tackled this aspect more carefully. The ACs decided to accept the paper. However, the authors should take the reviews and comments seriously when revising the paper for the camera ready.

(optional) Comment on this decision.


* the proposed fine-tuning of only the last layer is not novel enough
* experiments are not sufficient to isolate the differences to support the benefit of post-training

(optional) Comment on this decision.


This work is interested in using sentence vector representations as a method for both doing extractive summarization and as a way to better understand the structure of vector representations. While the methodological aspects utilize representation learning, the reviewers felt that the main thrust of the work would be better suited for a summarization workshop or even NLP venue, as it did not target DL based contributions. Additionally they felt that the work did not significantly engage with the long literature on the problem of summarization.

(optional) Comment on this decision.


This clearly written paper extends the Kronecker-factored approximate curvature optimizer to recurrent networks.  Experiments on Penn Treebank language modeling and training of differentiable neural computers on a repeated copy task show that the proposed K-FAC optimizers are stronger than SGD, Adam, and Adam with layer normalization. The most negative reviewer objected to a lack of theoretical error bounds on the approximations made, but the authors successfully argue that obtaining such bounds would require making assumptions that are likely to be violated in practice, and that strong empirical performance on real tasks is sufficient justification for the approximations.

Pros:
+ "Completes" K-FAC training by extending it to recurrent models.
+ Experiments show effects of different K-FAC approximations.

Cons:
- The algorithm is rather complex to implement.

(optional) Comment on this decision.


This paper generally presents a nice idea, and some of the modifications to searn/lols that the authors had to make to work with neural networks are possibly useful to others. Some weaknesses exist in the evaluation that everyone seems to agree on, but disagree about importance (in particular, comparison to things like BLS and Mixer on problems other than MT).

A few side-comments (not really part of meta-review, but included here anyway):
- Treating rollin/out as a hyperparameter is not unique to this paper; this was also done by Chang et al., NIPS 2016, "A credit assignment compiler..."
- One big question that goes unanswered in this paper is "why does learned rollin (or mixed rollin) not work in the MT setting." If the authors could add anything to explain this, it would be very helpful!
- Goldberg & Nivre didn't really introduce the _idea_ of dynamic oracles, they simply gave it that name (e.g., in the original Searn paper, and in most of the imitation learning literature, what G&n call a "dynamic oracle" everyone else just calls an "oracle" or "expert")

(optional) Comment on this decision.


This paper presents an unsupervised GAN-based model for disentagling the multiple views of the data and their content.

Overall it seems that this paper was well received by the reviewers, who find it novel and significant . The consensus is that the results are promising.

There are some concerns, but the major ones listed below have been addressed in the rebuttal. Specifically:
-	R3 had a concern about the experimental evaluation, which has been addressed in the rebuttal.
-	R2 had a concern about a problem inherent in this setting (what is treated as “content”), and the authors have clarified in the discussion the assumptions under which such methods operate.
-	R1 had concerns related to how the proposed model fits in the literature. Again, the authors have addressed this concern adequately.

(optional) Comment on this decision.


Three reviewers recommended rejection, and there was no rebuttal.

(optional) Comment on this decision.


The submission proposes a loss surrogate for top-k classification, as in the official imagenet evaluation.  The approach is well motivated, and the paper is very well organized with thorough technical proofs in the appendix, and a well presented main text.  The main results are: 1) a theoretically motivated surrogate, 2) that gives up to a couple percent improvement over cross-entropy loss in the presence of label noise or smaller datasets.

It is a bit disappointing that performance is limited in the ideal case and that it does not more gracefully degrade to epsilon better than cross entropy loss.  Rather, it seems to give performance epsilon worse than cross-entropy loss in an ideal case with clean labels and lots of data.  Nevertheless, it is a step in the right direction for optimizing the error measure to be used during evaluation.  The reviewers uniformly recommended acceptance.

(optional) Comment on this decision.


The paper proposes to study the impact of normalizing the gradient for each layer before applying existing techniques such as SG + momentum, Adam or AdaGrad. The study is done on a reasonable number of datasets and, after the reviewers' comments, confidence intervals have been added,  although Table 1 puts results in bold but many of these results are not statistically significant.

The paper, however, lacks a proper analysis of the results. Two main things could be improved:
- Normalization does not always have the same effect but the reasons for it are not discussed. This needs not be done theoretically but a more thorough analysis would have been appreciated.
- There is no hyperparameter tuning, which means that the results are heavily dependent on which hyperparameters were chosen. Thus, it is hard to draw any conclusion.

Regarding the seemingly conflicting remarks of the two reviewers, it all depends on what the paper is trying to achieve. If it tries to show that is it state-of-the-art, then comparing to state-of-the-art algorithms on every dataset is crucial. If it tries to study the impact of one specific change, in this case layer normalization, on the optimization, then comparing to the vanilla version is fine. The paper seems to try to address the latter so it is OK if it is not compared to all the state-of-the-art algorithms. However, proper tuning of existing methods is still required.

Ultimately, a better understanding of layer normalization could be useful but the paper is not convincing enough to provide that understanding. There is no need to increase the number of datasets but it should rather focus on designing setups to test and validate hypotheses.

(optional) Comment on this decision.


The proposed neural tree transduction framework is basically a combination of tree encoding and tree decoding. The tree encoding component is simply reused from previous work (TreeLSTM) whereas the decoding components is somewhat different from the previous work. They key problems (acknowledge also by at least 2 reviewers):

Pros:
-- generating trees input under-explored direction (note that it is more general than parsing as nodes may not directly correspond to input symbols)

Cons:
-- no comparison with previous tree-decoding work
-- only artificial experiments
-- the paper is hard too read (confusing) / mathematical notation and terminology is confusing and seems sometimes inaccurate (see R3)

(optional) Comment on this decision.


The paper proposes a new analysis of the optimization method called entropy-sgd which seemingly leads to more robust neural network classifiers. This is a very important problem if successful. The reviewers are on the fence with this paper. On the one hand they appreciate the direction and theoretical contribution, while on the other they feel the assumptions are not clearly elucidated or justified. This is important for such a paper. The author responses have not helped in alleviating these concerns. As one of the reviewers points out, the writing needs a massive overhaul. I would suggest the authors clearly state their assumptions and corresponding justifications in future submissions of this work.

(optional) Comment on this decision.


The AIRL is presented as a scalable inverse reinforcement learning algorithm. A key idea is to produce "disentangled rewards", which are invariant to changing dynamics; this is done by having the rewards depend only on the current state. There are some similarities with GAIL and the authors argue that this is effectively a concrete implementation of GAN-GCL that actually works.  The results look promising to me and the portability aspect is neat and useful!

In general, the reviewers found this paper and its results interesting and I think the rebuttal addressed many of the concerns. I am happy that the reproducibility report is positive which helped me put this otherwise potentially borderline paper into the 'accept' bucket.

(optional) Comment on this decision.


The paper presents a domain-specific language for RNN architecture search, which can be used in combination with learned ranking function or RL-based search. While the approach is interesting and novel, the paper would benefit from an improved evaluation, as pointed out by reviewers. For example, the paper currently evaluated coreDSL+ranking for language modelling and extendedDSL+RL for machine translation. The authors should use the same evaluation protocol on all tasks, and also compare with the state-of-the-art MT approaches.

(optional) Comment on this decision.


The novelty of the paper is limited and it lacks on comparisons with relevant baselines, as pointed out by the reviewers.

(optional) Comment on this decision.


I'm inclined to recommend accepting this paper, although it is borderline given the strong dissenting opinion. The revisions have addressed many of the concerns about quality, clarity, and significance. The paper gives an end to end explanation in Bayesian terms of generalization in neural networks using SGD.

However, it is my opinion that Bayesian statistics is not, at present, a theory that can be used to explain why a learning algorithm works. The Bayesian theory is too optimistic: you introduce a prior and model and then trust both implicitly. Relative to any particular prior and model (likelihood), the Bayesian posterior is the optimal summary of the data, but if either part is misspecified, then the Bayesian posterior carries no optimality guarantee. The prior is chosen for convenience here. And the model (a neural network feeding into cross entropy) is clearly misspecified.

However, there are ways to sidestep both these issues using a frequentist theory closely related to Bayes, which can explain generalization. Indeed, you cite a recent such paper by Dzugate and Roy who use PAC-Bayes. However, you citation is disappointingly misleading: a reader would never know that these authors are also responding to Zhang, have already proposed to explain "broad minima" in (PAC-)Bayesian terms, and then even get nonvacuous bounds. (The connection between PAC-bayes and marginl likelihood is explained by Germain et al. "PAC-Bayesian Theory Meets Bayesian Inference").  Dzugate et al don't propose to explain why SGD finds such "good" minima. So I would say, your work provides the missing half of their argument. This work deserves more prominent placement and shouldn't be buried on page 5. Indeed, it should appear in the introduction and a proper description of the relationship should be given.

(optional) Comment on this decision.


The experimental work was seen as one of the main weaknesses.

(optional) Comment on this decision.